Extending ASP.NET MVC with custom binders and results

Recently, I was putting together a brief sample for someone who wanted to use protobuf-net with ASP.NET as part of a basic HTTP transport; fairly simple stuff – just using protobuf as the binary body of the HTTP request/response.

It was a simple enough demo, but it got me wondering: what would be required to do this (more cleanly) in ASP.NET MVC? In many ways we just want something pretty similar to how many people currently use JSON: forget the formal API – just tell me the routes and let me know what data I need.

It turns out that this is astonishingly easy…

The controller

A good way of understanding the big picture is to look at the controller. So let’s take a peek:

One very basic controller, with one action. Things to note:

• We want to read the request parameter (“req”) from the request body
• To do that, we’re using a custom binder, via [ProtoPost] which we’ll see shortly
• We could have exposed different input arguments separately – it would have added complexity, and isn’t very compatible with the way that requests are written in the .proto language, so I’ve restricted myself to binding a single (composite) parameter
• We run some logic, same as normal
• We return an action-result that will write the response back as protobuf

The nice thing about this is that (no matter which option you use to control your routes) it is very simple to add actions (as different urls).

That might be all you need to know!

I’ll include more explanation for completeness, but if you want a simple way of throwing objects over http efficiently and in an interoperable way, that might be enough.

The binder

The binder’s job is it understand the incoming request, and map that into things the controller can understand, such as method variables. There are two parts to a binder: writing the code to handle the input, and telling the system to use it. Writing a binder, it turns out, can be pretty simple (well – if you already have the serialization engine available); here’s all there is:

This simply checks that the caller is doing a POST, and if they are it passes the input stream to protobuf-net, based on the parameter type (which is available from the binding-context).

The other part is to tell ASP.NET MVC to use it; in my case I’m using a new [ProtoPost] attribute which simply creates the binder:

The result

Just as a controller doesn’t need to know how to parse input, it also doesn’t know how to format output – that is the job of an action-result; we’ll write our own, that we can re-use to write objects to the response stream. Which is actually less work than writing the last sentence!

Again – some very basic code that simply takes an object and writes it to the output stream via protobuf-net.

Anything else?

OK, I’ve taken some shortcuts there. To do it properly you might want to check for an expected content-type, etc – and things like security is left open for your own implementation (just like it would be for any other similar object-passing API). I’m just showing the core object <===> stream code here. But it works, and is painless.

The example code (include the ASP.NET / IHttpHandler equivalent, and an example client) is all available in the protobuf-net trunk, here.

protobuf-net; a status update

I keep promising “v2” is just around the corner, and (quite reasonably) I get a steady stream of “when” e-mails / tweets etc, so it is time for a status update…

…but first I must apologise for the delay; the short version is that in the last few weeks I’ve changed jobs, and it has taken a bit of time to get my daily pattern (work time, family time, geek time, etc) sorted. Since I no longer spend 4 hours a day commuting, in theory I have more time available. In reality I’m spending lots more time with my family, which I don’t regret. But I’m getting back into the “crazy geek” thing ;p

So where is protobuf-net?

The first thing to note that “v2” is not a minor change; it completely changes the crazy-complex generics pipeline (which killed CF, and was generally a bad design choice for lots of reasons), opting instead for a non-generic pipeline coupled with an IL emitter to get the maximum performance. It also introduces a whole new metadata abstraction layer, and static dll usage (so it works on iPhone (but don’t ask me how the SDK rules affect this; IANAL), Phone 7, and so that CF doesn’t have to use reflection).

The numbers

• Tests for new new features (metadata abstraction, IL generation, etc): 100%
• Compatibility tests:79% (228 / 287, plus some skipped that won’t be in the first alpha release)

I should note: my unit tests for this tend, right or wrong, to each cover a number of features; it is very hard to make the regression tests overly granular, but to be honest I think we can all agree that the main objective is not to have the world’s most beautiful tests, but rather: to check that the system works robustly.

The gaps

• Mapped enums; it’ll handle enums, but it currently only treats them as pass-thrus to the underlying primitive value. It doesn’t currently process [ProtoEnum]. Not hard, just time.
• Member inference (which was the best “v1” could do to allow serialization of types outside your control); this doesn’t involve IL, so not too tricky – and not used by the majority.
• Roundtrip for unexpected fields (currently dropped; this also impacts the memcached transcoder)
• Parse/ToString fallback for unknown types (another “v1” feature)
• Packed arrays (an optional, alternative wire-format for lists/arrays; this won’t affect you if you haven’t heard of it, since it is opt-in)
• *WithLengthPrefix – for reading/writing multiple objects to/from a single stream
• GetProto (this is deferred from the alpha release, as it isn’t “core” functionality, and is significant work to do properly)

No show-stoppers; just work to do. Not long now. I’m reluctant to promise a “when” since I’ve delayed already…

When to optimize?

Prompted by a tweet earlier, I thought I’d share some thoughts on optimization. In code, there are diverging views here, varying between the extremes:

• Forget the performance; write an awesome application, then see (perhaps profile, perhaps just use it) where it sucks
• Performance rules! Give yourself an ulcer trying to eek out every last picosecond of CPU and nibble of RAM!

Obviously there is some pragmatic middle-ground in the middle where you need to design an app deliberately to allow performance, without stressing over every line of code, every stack-frame, etc.

Is this juxtaposition valid?

I suspect it is. In my day job I’m an app developer: what matters is shipping an app that works, is usable, is clear, doesn’t corrupt the database, doesn’t e-mail real clients from the test server*, etc. Performance is important, but management aren’t going to be impressed with “I shaved 3ms/cycle from the ‘frobing’ loop! oh, yes, but the login page is still broken”.

But my “for kicks” OSS alter-ego is a library developer. Shipping your library with great features is important, but library developers are (in my experience at least) far more interested in all the micro-optimisations. And this makes sense; the people using your library are trusting you to do it well – they’re too busy getting the login page working. And for a library developer, getting these details right is part of how you deliver the best product.

Summary

So next time you hear somebody worrying about whether to use string.Concat(string[]), string.Concat(object[]) or StringBuilder: it could be random micro-optimisation gone wild (and please do kick them); but maybe, just maybe, you’re dealing with a closet library developer.

*=yes, I’ve accidentally done that. Once. Fortunately we caught it quickly, and the data wasn’t rude or anything. Lawsuit avoided.

Strings: sorted

Sorting data… I don’t mean complex bespoke types that you’ve written – I mean inbuilt types like “string”. You probably wouldn’t think too long about trying to sort strings – you just expect them to work.

Oddly enough, this wasn’t actually always the case. OK, it is an edge case, but string comparison wasn’t actually guaranteed to be transitive. By this I mean that if we’ve got three values A, B and C, and we’ve tested that A comes before B, and B comes before C – then you might reasonably deduce that A comes before C. This is a key requirement for sorting (and has always been documented as such). But it wasn’t always the case!

Why is this bad? At best case, your data can choose random-looking sort orders. At worst case a “sort” operation may loop forever, attempting to shuffle data that is teasing it.

Even though this oddity was rare, it was problem enough for it to come up in the wild, way back when usenet was in vogue (in other news: Microsoft is shutting down their usenet farm… is that news to anyone?).

It was news to me, but while discussing this curio, somebody observed that it is now fixed in .NET 4.0; the nature of the fix means that the code in the connect article (above) needs re-ordering to show it working and not-working as you flip between framework versions; here’s the updated sample:

        string s1 = "-0.67:-0.33:0.33";
      string s2 = "-0.67:0.33:-0.33";
      string s3 = "0.67:-0.33:0.33";
    
      Console.WriteLine(s1.CompareTo(s2));
      Console.WriteLine(s2.CompareTo(s3));
      Console.WriteLine(s1.CompareTo(s3));

In .NET < 4.0 this shows “-1 –1 1” (this is a fail). In .NET 4.0, it shows “1 1 1” (a pass). I tried a range of combinations and couldn’t make it misbehave. A small matter, maybe, but I’m much happier knowing that this has finally been tucked up in bed.

Walkthrough: protobuf-net on Phone 7

UPDATE: better tools for this now exist; see precompile

Now’s my chance to demonstrate the (pre-alpha) “v2” protobuf-net API, while at the same time showing it working on Phone 7. In particular, this shows the new pre-compilation functionality, which is vital for good performance on Phone 7, iPhone, XNA, Compact Framework, etc (since they have scant meta-programming).

But first…

Why? Just why?

You have a mobile device; data should be considered a limited resource, both in terms of financial cost (data plans etc) and bandwidth-related performance. Therefore we want to send less over the wire.

• Google’s protobuf wire-format is extremely dense. In theory you can get a few bytes tighter if you squeeze every nibble manually, but you’ll go crazy in the process.
• Mobile devices rarely have full meta-programming; we don’t want reflection (too slow), so we want fully pre-compiled serializers.
• Code-generation is one option, but most code-gen APIs want you to use their object model. Thanks very much, but I already have an object model. I’ll use that, ta.
• Shiny.

Building your domain model

You can use protobuf-net from a .proto (Google’s DSL for describing protobuf schemas), but it is my belief that most .NET developers want to work with their existing model. So let’s start with a basic model.

For this demo, I’m starting with a Silverlight library project to represent my model (or it could also be a DTO layer). There are various ways of setting this up, but this is the simplest while I work out the kinks…

My model is very basic; just an order header / detail pair. For simplicity I’m using the protobuf-net attributes to allocate field-numbers to the properties, but there are may other ways of doing this now (including support for vanilla objects with no adornments). Here we go; very simple:

Generating a serialization assembly

This is the most obvious difference in “v2”; you can now pre-generate the serialization code into a fully static-typed assembly. The “generate at runtime” model is still supported for “regular” .NET, and has also had a complete overhaul improving performance noticeably.

“The plan” here is to write a simply utility exe that you can call in your build process to do this step, but I don’t have that complete yet. Instead, just for this demo, I’m going to use “regular” .NET console exe, referencing the “regular” protobuf-net assembly (and the DTO assembly), with an entire 4 lines of code:

This tells the API what types we are interested in, and offers a wide range of options for changing how the model is mapped. We’re passing in “true” to let it figure out the details itself.

Running this exe generates “MySerializer.dll”, which is our serialization assembly with a class “OrderSerializer”. It still needs some utility code from protobuf-net, so in our Phone 7 app we’ll need references to the DTO assembly, the serializer assembly and the Phone 7 version of protobuf-net.

 

Now all we need to do is use the serializer! In “v1” we would have used static methods on “ProtoBuf.Serializer”, but instead we want to use our specific pre-generated type:

We also need to create some sample data to play with:

Next, purely for illustration we’ll ignore the handy “DeepClone” method, and serialize / deserialize the object via a MemoryStream:

And because I have extremely limited UI skills, I’ll just throw the cloned data into a ListBox:

Et voila; working protobuf-net serialization via a pre-generated serialization dll, on Phone 7.

Initial performance tests seem to indicate that it is in the region of 30-50 times faster than the DataContractSerializer that ships with Phone 7, and the protobuf wire format typically takes about 20% of the bandwidth. Win win.

Caveats

The “v2” code isn’t complete yet! I’m still working through all the compatibility tests. It is getting there, though. It is a week or so away from a stable beta, I suspect.

Additionally, I’m still working through some of the issues associated with using “regular” .NET to generate the assembly for a light framework such as CF / Phone 7. Some combinations seem to work better than others.

protobuf-net v2 on Phone 7

I mentioned previously how protobuf-net “v2” works on iPhone; well, I guess we’re all still waiting to see how §3.3.1 plays out there…

So rather than mope, here it is working on Phone 7 Series emulator (¿simulator?), from the new CTP released today.

Importantly, Phone 7 lives somewhere between Silverlight and Compact Framework. It does not have any meta-programming support (no ILGenerator etc). So in “v1” (the currently downloadable dll) it had to fall back to reflection – slow, but it will get there.

In “v2”, we get the best of both worlds; on “full” .NET wecan use runtime meta-programming to generate fast IL on the fly, but if we know we are targeting a slim framework (or we just don’t want to use meta-programming at runtime) we can pre-generate a serialization dll.

I wish the screenshot was more exciting, but I’m a bit busy working through the last few “v1” compatibility checks. For obvious reasons I haven’t tried this on a physical device. When they’re ready, if Microsoft want to send me a unit for testing purposes, that would be fine… [holds breath].

Strong Naming on Phone 7

I spent a merry lunchtime looking at the new CTP of Phone 7. It is looking very promising, but interestingly it doesn’t (yet?) include options in the UI to strong-name your library projects. It does, however, still seem to work if you just edit the csproj by hand.

Notepad will do the job, or in Visual Studio right-click in Solution Explorer to “Unload Project”, then right-click again to “Edit …” it.

Add the lines below, and finally right-click to “Reload Project”. Inelegant, but I can live with it ;-p

<SignAssembly>true</SignAssembly>

<AssemblyOriginatorKeyFile>your.snk</AssemblyOriginatorKeyFile>

I have no idea if this is a sane thing to do, but it makes my code work (I’m doing something a bit out of the ordinary), so I’m happy.