Building a simple PicPaste replacement using Azure Web Apps and WebJobs

This post was supposed to be an introduction to Azure WebJobs, but it took a weird turn somewhere and became a guide to building a simple PicPaste replacement using just a wimple Azure Web App and a WebJob.

As such, it might not be a really useful app, but it does show how simple it is to build quite powerful things using Azure.

So, what is the goal? Well, the goal is to build a website that you can upload images to, and then get a simple Url to use when sharing the image. This is not complicated, but as I want to resize the image, and add a little overlay to it as well before giving the user the Url, I might run into performance issues if it becomes popular. So, instead I want the web app to upload the image to blob storage, and then have a WebJob process it in the background. Doing it like this, I can limit the number of images that are processed at the time, and use a queue to handle any peaks.

[note]Note: Is this a serious project? No, not really. Does it have some performance issues if it becomes popular? Yes. Did I build it as a way to try out background processing with WebJobs? Yes… So don’t take it too serious.

The first thing I need is a web app that I can use to upload images through. So I create a new empty ASP.NET project, adding support for MVC. Next I add a NuGet packages called WindowsAzure.Storage. And to be able to work with Azure storage, I need a new storage account. Luckily, that is as easy as opening the “Server Explorer” window, right-clicking the “Storage” node, and selecting “Create Storage Account…”. After that, I am ready to start building my application.

Inside the application, I add a single controller called HomeController. However, I don’t want to use the default route configuration. instead I want to have a nice, short and simple route that looks like this

routes.MapRoute(
name: "Image",
url: "{id}",
defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional }
);

Ok, now that that is done, I add my “Index” view, which is ridiculously simple, and looks like this

<!DOCTYPE html>

<html>
<head>
<meta name="viewport" content="width=device-width" />
<title>AzurePaste</title>
</head>
<body>
@using (Html.BeginForm("Index", "Home", FormMethod.Post, new { enctype = "multipart/form-data" }))
{
<label>Chose file: </label>
<input type="file" name="file" accept="image/*"/><br/>
<input type="submit" value="Upload" />
}
</body>
</html>

As you can see, it contains a simple form that allows the user to post a single file to the server using the Index method on the HomeController.

The only problem with this is that there’s no controller action called Index that accepts HTTP POSTs. So let’s add one.

[HttpPost]
public ActionResult Index(HttpPostedFileBase file)
{
if (file == null)
{
return new HttpStatusCodeResult(HttpStatusCode.BadRequest);
}

var id = GenerateRandomString(10);
var blob = StorageHelper.GetUploadBlobReference(id);
blob.UploadFromStream(file.InputStream);

StorageHelper.AddImageAddedMessageToQueue(id);

return View("Working", (object)id);
}

So what does this action do? Well, first of all it returns a HTTP 400 if you forgot to include the file… Next it uses a quick and dirty helper method called GenerateRandomString(). It just generates a random string with the specified length… Next, I use a helper class called StorageHelper, which I will return to shortly, to get a CloudBockBlob instance to which I upload my file. The name of the blob is the random string I just retrieved, and the container for it is a predefined one that the WebJob knows about. The WebJob will then pick it up there, make the necessary tranformations, and then save it to the root container.

Note: By adding a container called $root to a blob storage account, you get a way to put files in the root of the Url. Otherwise you are constrained to a Url like https://[AccountName].blob.core.windows.net/[ContainerName]/[FileName]. But using $root, it can be reduced to https://[AccountName].blob.core.windows.net/[FileName].

Once the image is uploaded to Azure, I once again turn to my StorageHelper class to put a message on a storage queue.

Note: I chose to build a WebJob that listened to queue messages instead of blob creation, as it can take up to approximately 10 minutes before the WebJob is called after a blob is created. The reason for this is that the blob storage logs are buffered and only written approximately every 10 minutes. By using a queue, this delay is decreased quite a bit. Bit it is still not instantaneous… If you need to decrease it even further, you can switch to a ServiceBus queue instead.

Ok, if that is all I am doing, that StorageHelper class must be really complicated. Right? No, not really. It is just a little helper to keep my code a bit more DRY.

The StorageHelper has 4 public methods. EnsureStorageIsSetUp(), which I will come back to, GetUploadBlobReference(), GetRootBlobReference() and AddImageAddedMessageToQueue(). And they are pretty self explanatory. The two GetXXXBlobReference() methods are just helpers to get hold of a CloudBlockBlob reference. By keeping it in this helper class, I can keep the logic of where blobs are placed in one place… The AddImageAddedMessageToQueue() adds a simple CloudQueueMessage, containing the name of the added image, on a defined queue. And finally, the EnsureStorageIsSetUp() will make sure that the required containers and queues are set up, and that the root container has read permission turned on for everyone.

public static void EnsureStorageIsSetUp()
{
UploadContainer.CreateIfNotExists();
ImageAddedQueue.CreateIfNotExists();
RootContainer.CreateIfNotExists();
RootContainer.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Blob });
}

public static CloudBlockBlob GetUploadBlobReference(string blobName)
{
return UploadContainer.GetBlockBlobReference(blobName);
}

public static CloudBlockBlob GetRootBlobReference(string blobName)
{
return RootContainer.GetBlockBlobReference(blobName);
}

public static void AddImageAddedMessageToQueue(string filename)
{
ImageAddedQueue.AddMessage(new CloudQueueMessage(filename));
}

Kind of like that…

The returned view that the user gets after uploading the file looks like this

@model string

<!DOCTYPE html>

<html>
<head>
<meta name="viewport" content="width=device-width" />
<title>Azure Paste</title>
<script src="~/Content/jquery.min.js"></script>
<script type="text/javascript">
$(function () {
function checkCompletion() {
setTimeout(function () {
$.ajax({ url: "/@Model" })
.done(function (data, status, jqXhr) {
if (jqXhr.status === 204) {
checkCompletion();
return;
}
document.location.href = data;
})
.fail(function (error) {
alert("Error...sorry about that!");
console.log("error", error);
});
}, 1000);
}
checkCompletion();
});
</script>
</head>
<body>
<div>Working on it...</div>
</body>
</html>

As you can see, the bulk of it is some JavaScript, while the actual content that the users sees is really tiny…

The JavaScript uses jQuery (no, not a fan, but it has some easy ways to do ajax calls) to poll the server every second. It calls the server at “/[FileName]”, which as you might remember from my changed routing, will call the Index method on the HomeController.

If the call returns an HTTP 204, the script keeps on polling. If it returns HTTP 200, it redirects the user to a location specified by the returned content. If something else happens, if just alerts that something went wrong…

Ok, so this kind of indicates that my Index() method needs to be changed a bit. It needs to do something different if the id parameter is supplied. So I start by handling that case

public ActionResult Index(string id)
{
if (string.IsNullOrEmpty(id))
{
return View();
}

...
}

That’s pretty much the same as it is by default. But what if the id is supplied? Well, then I start by looking for the blob that the user is looking for. If that blob exists, I return an HTTP 200, and the Url to the blob.

public ActionResult Index(string id)
{
...

var blob = StorageHelper.GetRootBlobReference(id);
if (blob.Exists())
{
return new ContentResult { Content = blob.Uri.ToString().Replace("/$root", "") };
}

...
}

As you can see, I remove the “/$root” part of the Url before returning it. The Azure Storage SDK will include that container name in the Url even if it is a “special” container that isn’t needed in the Url. So by removing it I get this nicer Url.

If that blob does not exist, I look for the temporary blob in the upload folder. If it exists, I return an HTTP 204. And if doesn’t, then the user is looking for a file that doesn’t exist, so I return a 404.

public ActionResult Index(string id)
{
...

blob = StorageHelper.GetUploadBlobReference(id);
if (blob.Exists())
{
return new HttpStatusCodeResult(HttpStatusCode.NoContent);
}

return new HttpNotFoundResult();
}

Ok, that is all there is to the web app. Well…not quite. I still need to ensure that the storage stuff is set up properly. So I add a call to the StorageHelper.EnsureStorageIsSetUp() in the Application_Start() method in Global.asax.cs.

protected void Application_Start()
{
AreaRegistration.RegisterAllAreas();
RouteConfig.RegisterRoutes(RouteTable.Routes);

StorageHelper.EnsureStorageIsSetUp();
}

Next up is the WebJob that will do the heavy lifting. So I add a WebJob project to my solution. This gives me a project with 2 C# files. One called Program.cs, which is the “entry point” for the job, and one called Functions.cs, which contains a sample WebJob.

The first thing I want to do is to make sure that I don’t overload my machine by running too many of these jobs in parallel. Doing image manipulation is hogging resources, and I don’t want it to get too heavy for the machine.

I do this by setting the batch size for queues in a JobHostConfiguration inside the Program.cs file

static void Main()
{
var config = new JobHostConfiguration();
config.Queues.BatchSize = 2;

var host = new JobHost(config);
host.RunAndBlock();
}

Now that that is done, I can start focusing on my WebJob… I start out by deleting the existing sample job, and add in a new one.

A WebJob is just a static method with a “trigger” as the first parameter. A trigger is just a method parameter that has an attribute set on it, which defines when the method should be run. In this case, I want to run it based on a queue, so I add a QueueTrigger attribute to my first parameter.

As my message contains a simple string with the name of the blob to work on, I can define my first parameter as a string. Had it been something more complicated, I could have added a custom type, which the calling code would have populated by deserializing the content in the message. Or, I could have chosen to go with CloudQueueMessage, which gives med total control. But as I said, just a string will do fine. It will also help me with my next  three parameters.

As a lot of WebJobs will be working with Azure storage, the SDK includes helping tools to make this easier. One such tool is an attribute called BlobAttribute. This makes it possible to get a blob reference, or its contents, passed into the method. In this case, getting references to the blobs I want to work with makes things a lot easier. I don’t have to handle getting references to them on my own. All I have to do, is to add parameters of type ICloudBlob, and add a BlobAttribute to them. The attribute takes a name-pattern as the first string. But in this case, the name of the blob will be coming from the queue message… Well, luckily, the SDK people have thought of this, and given us a way to access this by adding “{queueTrigger}” to the pattern. This will be replaced by the string in the message…

Ok, so the signature fore my job method turns in to this

public static void OnImageAdded([QueueTrigger("image-added")]string blobName,
[Blob("upload/{queueTrigger}")]ICloudBlob blob,
[Blob("$root/{queueTrigger}.png")]ICloudBlob outputImage,
[Blob("$root/{queueTrigger}")]ICloudBlob htmlBlob,
TextWriter log) {
...
}

As you can see, I have also added a final parameter of type TextWriter, called log. This will be a log supplied by the WebJob host, making it possible to log things in a nice a uniform way.

Bu wait a little…why am I taking in 3 blobs? The first one is obviously the uploaded image. The second one is the target where I am going to put the transformed image. What is the last one? Well, I am going to make it a little more complicated than just hosting the image… I am going to host the image under a name which is [UploadedBlobName].png. I am then going to add a very simple HTML file to show the image in a blob at the same name as the uploaded blob. That ay, the Url to the page to view the image will be a nice and simple one, and it will show the image and a little text.

The first thing I need to do is get the content of the blob. This could have been done by requesting a Stream instead of a IClodBlob, but as I want to be able to delete it at the end, that didn’t work…unless I used more parameters, which felt unnecessary…

Once I have my stream, I turn it into a Bitmap class from the System.Drawing assembly. Next, I resize that image to a maximum with or height before adding a little watermark.

public static void OnImageAdded([QueueTrigger("image-added")]string blobName,
[Blob("upload/{queueTrigger}")]ICloudBlob blob,
[Blob("$root/{queueTrigger}.png")]ICloudBlob outputImage,
[Blob("$root/{queueTrigger}")]ICloudBlob htmlBlob,
TextWriter log)
{
var ms = new MemoryStream();
blob.DownloadToStream(ms);

var image = new Bitmap(ms);
var newImage = image.Resize(int.Parse(ConfigurationManager.AppSettings["AzurePaste.MaxSize"]));
newImage.AddWatermark("AzurePaste FTW");

...
}

Ok, now that I have my transformed image, it is time to add it to the “target blob”. I do this by saving the image to a MemoryStream and then uploading that. However, by default, all blobs get the content type “application/octet-stream”, which isn’t that good for images. So I update the blob’s content type to “image/png”.
public static void OnImageAdded([QueueTrigger("image-added")]string blobName,
[Blob("upload/{queueTrigger}")]ICloudBlob blob,
[Blob("$root/{queueTrigger}.png")]ICloudBlob outputImage,
[Blob("$root/{queueTrigger}")]ICloudBlob htmlBlob,
TextWriter log)
{
...

var outputStream = new MemoryStream();
newImage.Save(outputStream, ImageFormat.Png);
outputStream.Seek(0, SeekOrigin.Begin);
outputImage.UploadFromStream(outputStream);
outputImage.Properties.ContentType = "image/png";
outputImage.SetProperties();

...
}

The last part is to add the fabled HTML page… In this case, I have just quickly hacked a simple HTML page into my assembly as an embedded resource. I could of course have used one stored in blob storage or something, making it easier to update. But I just wanted something simple…so I added it as an embedded resource… But before I add it to the blob, I make sure to replace the Url to the blob, which has been defined as “{0}” in the embedded HTML.

public static void OnImageAdded([QueueTrigger("image-added")]string blobName,
[Blob("upload/{queueTrigger}")]ICloudBlob blob,
[Blob("$root/{queueTrigger}.png")]ICloudBlob outputImage,
[Blob("$root/{queueTrigger}")]ICloudBlob htmlBlob,
TextWriter log)
{
...

string html;
using (var htmlStream = typeof (Functions).Assembly.GetManifestResourceStream("DarksideCookie.Azure.WebJobs.AzurePaste.WebJob.Resources.page.html"))
using (var reader = new StreamReader(htmlStream))
{
html = string.Format(reader.ReadToEnd(), blobName + ".png");
}
htmlBlob.UploadFromStream(new MemoryStream(Encoding.UTF8.GetBytes(html)));
htmlBlob.Properties.ContentType = "text/html";
htmlBlob.SetProperties();

blob.Delete();
}

As you can see, I also make sure to update the content type of this blob to “text/html”. And yeah…I also make sure to delete the original file.

That is about it… I guess I should mention that the Resize() and AddWatermark() methods in the Bitmap are extension methods I have added. They are not important for the topic, so I will just leave them out. But they are available in the downloadable code below.

There is one more thing though… What happens if my code is borked, or someone uploads a file that isn’t an image? Well, in that case, the code will fail, and fail, and fail… If it fails, the WebJob will be re-run at a later point. Unfortunately, this can turn ugly, and turn into what is known as a poison message. Luckily, this is handled by default. After a configurable amount of retries, the message is considered a poison message, and will be discarded. As it is, a new message is automatically added to a dynamically created queue to notify us about it. So it might be a good idea for us to add a quick little job to that queue as well, and log any poison messages.

The name of the queue that is created is “[OriginalQueueName]-poison”, and the handler for it looks like any other WebJob. Just try not to add code in here that turns these messages into poison messages…

public static void LogPoisonBlob([QueueTrigger("image-added-poison")]string blobname, TextWriter logger)
{
logger.WriteLine("WebJob failed: Failed to prep blob named {0}", blobname);
}

That’s it! Uploading an image through the web app will now place it in blob storage. it will then wait until the WebJob has picked it up, transformed it, and stored it, and a new html file, at the root of the storage account. Giving the user a quick and easy address to share with his or her friends.

Note: If you want to run something like this in the real world, you probably want to add some form of clean-up solution as well. Maybe a WebJob on a schedule, that removes any images older than a certain age…

And as usual, there is a code sample to play with. Just remember that you need to set up a storage account for it, and set the correct connectionstrings in the web.config file. As well as the WebJob project’s app.config if you want to run it locally.

Note: It might be good to know that logs and information about current wWebJobs can be found at https://[ApplicationName].scm.azurewebsites.net/azurejobs/#/jobs

Source code: DarksideCookie.Azure.WebJobs.AzurePaste.zip (52.9KB)

Cheers!

DotNetRocks with Carl Franklin and Richard Campbell

A while back, at NDC Oslo, I was approached by Carl Franklin and Richard Campbell regarding joining them for an episode of their podcast DotNetRocks. Obviously I said yes without hesitating! Who wouldn’t!? Unfortunately there were some scheduling problems, so we couldn’t do it during NDC. So instead, we did it over Skype a week or so later. And it has now finally been published on the DotNetRocks website.

The topic for the episode is the SOLID principles, which I have been talking about at quite a few conferences now. It was nice talking about these principles under these relaxed circumstances, without getting some of the flaming that I get ever so often for being a bit too pragmatic about the whole thing. Hopefully people will enjoy listening to it, and get some thoughts and ideas about how to work with SOLID in there projects. So go ahead and have a listen at http://bit.ly/dotnetrockssolid.

Integrating a front-end build pipeline in ASP.NET builds

A while back I wrote a couple of blog posts about how to set up a “front-end build pipeline” using Gulp. The pipeline handled things like less to css conversion, bundling and minification, TypeScript to JavaScript transpile etc. However, this pipeline built on the idea that you would have Gulp watching your files as you worked, and doing the work as they changed. The problem with this is that it only runs on the development machine, and not the buildserver.… I nicely avoided this topic by ending my second post with “But that is a topic for the next post”. I guess it is time for that next post…

So let’s start by looking at the application I will be using for this example!

More...

Integrating with Github Webhooks using OWIN

For some reason I got the urge to have a look at webhooks when using GitHub. Since it is a feature that is used extensively by build servers and other applications to do things when code is pushed to GitHub etc, I thought it might be cool to have a look at how it works under the hood. And maybe build some interesting integration in the future…

The basic idea behind it is that you tell GitHub that you want to get notified when things happen in your GitHub repo, and GitHub makes sure to do so. It does so using a regular HTTP call to an endpoint of your choice.

More...

Trying to understand the versioning “mess” that is .NET 2015

Right now there is a lot of talk about the next iteration of the .NET platform, and the different versions and runtimes that is about to be released. Unfortunately, it has turned into a quite complicated situation when it comes to the versions of things being released.

I get quite a few question about how it all fits together, and I try answering them as best as I can. However, as the question keeps popping up over and over again, I thought I would sum up the situation as I have understood it.

Disclaimer: Everything in this post is “as I have understood it”. I am not working for Microsoft, and I am in no way or form guaranteeing that this is the right description. This is just how I understand the situation. Hopefully it is fairly close to the real world.

More...

ASP.NET 5 demo code from SweNug

As promised during my talk at SweNug Stockholm last week, I have now uploaded my code for anyone to play around with. I just want to highlight that it is based around VERY early bits, and new versions might cause problems. It is built using runtime 1.0.0-beta5-11533. But even if it changes, the code should give you a clue about how it works.

Code is available here: ASP.NET 5 Demo.zip (714.47 kb)

Getting the ASP.NET 5 samples to work on Windows with the “new” dnvm (and beta 5)

[UPDATE] If you clone the dev branch’s samples instead of the master branch, it should be easier. You will still need to update the Startup.cs file for now though. Pull-request made… (That was a long blog post to no use…still gives some insight into how it works though…) [END UPDATE]

[UPDATE 2] Now updated to beta-5 if anyone still wants it considering the update above… [END UPDATE 2]

Yesterday I finally had time to sit down and play with the new ASP.NET 5 runtime, which is something I have wanted to do for quite some time. However, it kind of annoyed me that I couldn’t just install the ASP.NET 5 runtime, clone the samples from GitHub and get started, as it generated a whole heap of errors. After an hour or so of Googling and trying things out, I finally got it working, so I thought I would write down what I did to get it to work.

Note: This codebase is moving ridiculously fast, so this post is going to be old within a very short while. Everything is based on the code as of today, March 20th 2015.

More...

Understanding OWIN (Katana) Authentication Middleware

As some of you might have noticed, I really like OWIN. I like the simplicity, and the extremely powerful things that you can do with it in a simple way. And the fact that I don’t have to create an IHttpModule implementation, and figure out the ASP.NET event to hook into, like I had to to do the same thing before OWIN.

Katana, Microsoft’s implementation of OWIN, also offers a standardized way to handle authentication. And it is really easy to use, and not too hard to extend to work with your own identity providers. However, being me, I want to know how it works “under the hood”, and not just read a “how to build an authentication middlware” blog post…

Remember, knowing how things work “under the hood”, or “under the bonnet” if you speak the Queens English, makes it possible to do more things than just use it. By knowing how a combustion engine works (under the hood/bonnet of your car), makes it possible to add a turbo or two to it, or a compressor, or at maybe tweak the fuel consumption and horse power you get from it. But let’s leave the car analogies and look at Katana authentication middleware.

More...

Uploading files using ASP.NET Web Api

I have once again been tasked with writing a ASP.NET Wep Api action to handle the upload of a file. Or rather a form including a file. And once again, I Googled it looking for good solutions. And once again I got the same code thrown in my face over and over again. It seems to be the only one out there. And it isn’t what I want…

The code I am talking about is the one available at http://www.asp.net/web-api/overview/advanced/sending-html-form-data,-part-2. And if you don’t want to go and read that, the general gist is this

if (!Request.Content.IsMimeMultipartContent())
throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);

string root = HttpContext.Current.Server.MapPath("~/App_Data");
var provider = new MultipartFormDataStreamProvider(root);

await Request.Content.ReadAsMultipartAsync(provider);

/* Get files using provider.FileData */

More...

Building a “front-end build pipeline” from a C# devs perspective - Part 2

In the previous post, we looked at how we can use Gulp to run tasks for us. And in that post we used it to create tasks for transpiling LESS and TypeScript into CSS and JavaScript. But the example was very small and simple. It only contained 1 LESS file and 1 JavaScript file. But what if we have more than 1? Well, that’s when we need to start bundling the files together, and potentially minify them so that they are faster to download. Luckily, this is a piece of cake to do using Gulp. So in this post, we will have a look at how to do that, as well as how to get some TypeScript/JavaScript tests thrown in there as well.

Dislaimer: The solution will still be VERY small and simple. But at least it will be made big enough to be able to use bundling and minification. Which to be honest just means that we need more than one file of each type…

I assume that you have read the last post, and that if you are following along on your machine, you will need to be done with everything that was done in that post…

More...