Walkthrough: protobuf-net on Phone 7

UPDATE: better tools for this now exist; see precompile

Now’s my chance to demonstrate the (pre-alpha) “v2” protobuf-net API, while at the same time showing it working on Phone 7. In particular, this shows the new pre-compilation functionality, which is vital for good performance on Phone 7, iPhone, XNA, Compact Framework, etc (since they have scant meta-programming).

But first…

Why? Just why?

You have a mobile device; data should be considered a limited resource, both in terms of financial cost (data plans etc) and bandwidth-related performance. Therefore we want to send less over the wire.

• Google’s protobuf wire-format is extremely dense. In theory you can get a few bytes tighter if you squeeze every nibble manually, but you’ll go crazy in the process.
• Mobile devices rarely have full meta-programming; we don’t want reflection (too slow), so we want fully pre-compiled serializers.
• Code-generation is one option, but most code-gen APIs want you to use their object model. Thanks very much, but I already have an object model. I’ll use that, ta.
• Shiny.

Building your domain model

You can use protobuf-net from a .proto (Google’s DSL for describing protobuf schemas), but it is my belief that most .NET developers want to work with their existing model. So let’s start with a basic model.

For this demo, I’m starting with a Silverlight library project to represent my model (or it could also be a DTO layer). There are various ways of setting this up, but this is the simplest while I work out the kinks…

My model is very basic; just an order header / detail pair. For simplicity I’m using the protobuf-net attributes to allocate field-numbers to the properties, but there are may other ways of doing this now (including support for vanilla objects with no adornments). Here we go; very simple:

Generating a serialization assembly

This is the most obvious difference in “v2”; you can now pre-generate the serialization code into a fully static-typed assembly. The “generate at runtime” model is still supported for “regular” .NET, and has also had a complete overhaul improving performance noticeably.

“The plan” here is to write a simply utility exe that you can call in your build process to do this step, but I don’t have that complete yet. Instead, just for this demo, I’m going to use “regular” .NET console exe, referencing the “regular” protobuf-net assembly (and the DTO assembly), with an entire 4 lines of code:

This tells the API what types we are interested in, and offers a wide range of options for changing how the model is mapped. We’re passing in “true” to let it figure out the details itself.

Running this exe generates “MySerializer.dll”, which is our serialization assembly with a class “OrderSerializer”. It still needs some utility code from protobuf-net, so in our Phone 7 app we’ll need references to the DTO assembly, the serializer assembly and the Phone 7 version of protobuf-net.

 

Now all we need to do is use the serializer! In “v1” we would have used static methods on “ProtoBuf.Serializer”, but instead we want to use our specific pre-generated type:

We also need to create some sample data to play with:

Next, purely for illustration we’ll ignore the handy “DeepClone” method, and serialize / deserialize the object via a MemoryStream:

And because I have extremely limited UI skills, I’ll just throw the cloned data into a ListBox:

Et voila; working protobuf-net serialization via a pre-generated serialization dll, on Phone 7.

Initial performance tests seem to indicate that it is in the region of 30-50 times faster than the DataContractSerializer that ships with Phone 7, and the protobuf wire format typically takes about 20% of the bandwidth. Win win.

Caveats

The “v2” code isn’t complete yet! I’m still working through all the compatibility tests. It is getting there, though. It is a week or so away from a stable beta, I suspect.

Additionally, I’m still working through some of the issues associated with using “regular” .NET to generate the assembly for a light framework such as CF / Phone 7. Some combinations seem to work better than others.

protobuf-net v2 on Phone 7

I mentioned previously how protobuf-net “v2” works on iPhone; well, I guess we’re all still waiting to see how §3.3.1 plays out there…

So rather than mope, here it is working on Phone 7 Series emulator (¿simulator?), from the new CTP released today.

Importantly, Phone 7 lives somewhere between Silverlight and Compact Framework. It does not have any meta-programming support (no ILGenerator etc). So in “v1” (the currently downloadable dll) it had to fall back to reflection – slow, but it will get there.

In “v2”, we get the best of both worlds; on “full” .NET wecan use runtime meta-programming to generate fast IL on the fly, but if we know we are targeting a slim framework (or we just don’t want to use meta-programming at runtime) we can pre-generate a serialization dll.

I wish the screenshot was more exciting, but I’m a bit busy working through the last few “v1” compatibility checks. For obvious reasons I haven’t tried this on a physical device. When they’re ready, if Microsoft want to send me a unit for testing purposes, that would be fine… [holds breath].

Strong Naming on Phone 7

I spent a merry lunchtime looking at the new CTP of Phone 7. It is looking very promising, but interestingly it doesn’t (yet?) include options in the UI to strong-name your library projects. It does, however, still seem to work if you just edit the csproj by hand.

Notepad will do the job, or in Visual Studio right-click in Solution Explorer to “Unload Project”, then right-click again to “Edit …” it.

Add the lines below, and finally right-click to “Reload Project”. Inelegant, but I can live with it ;-p

<SignAssembly>true</SignAssembly>

<AssemblyOriginatorKeyFile>your.snk</AssemblyOriginatorKeyFile>

I have no idea if this is a sane thing to do, but it makes my code work (I’m doing something a bit out of the ordinary), so I’m happy.

Using dynamic to simplify reflection (and: generic specialization)

As I’ve noted in the past; reflection and generics don’t mix very well. The story isn’t quite as bad in .NET 4 though; here’s an example of something that is just painful to call (for the correct T known only at runtime) in C# 3.0:

And remember – we’re talking about the scenario where T is only known at runtime:

Previously we needed to use lots of MakeGenericMethod (or MakeGenericType) – things like:

You can see how that might quickly become boring. But in .NET 4 / C# 4.0 we can exploit “dynamic” to do all the heavy lifting and cache a delegate per scenario for us:

Much better!

Another possible use of this is generic type specialization:

Now this version is called for `int` arguments, otherwise the generic version is called – just as would have been the case from static C#. Magic!

The spec is a lie! (the cake is probably real)

My current bugbear: the C# 4.0 spec. Obviously, with a new toy like VS2010 the first thing you do is read all the documentation, yes? yes? anyone? Oh, just me then…

For whatever reason, the version that shipped in the release version of VS2010 isn’t the proper version. We can conclude this at a minimum because it doesn’t include the lock or event changes.

Now call me crazy, but I’m of the opinion that the spec is quite important in something like a programming language, yet at the moment the new shiny compiler is (by definition) not a compliant C# 4.0 compiler; it generates the wrong IL (according to the spec) for locks and field-like events.

I’m also still very unclear whether any other subtle changes have sneaked in (excluding the obvious changes for things like dynamic, variance, named arguments, optional parameters, etc).

I should stress: my aim here is in part to say “don’t bother re-reading the spec yet”, and in part to add one more grain of sand to the nagging I’ve been doing behind the scenes for ages on this topic. The right people already know about this – I just really want them to get it done and publish the darned spec! And ideally a change log, if they are feeling generous.

You know it is the right thing!

protobuf-net / VS2010 (and writing VS2010 packages)

With the release of VS2010, I was torn – between:

• re-releasing the current VS add-in (for protobuf-net v1, but with VS2010 support)
• holding off (building VS packages is fiddly) and releasing the (hopefully not-very-far-away-at-all-honest-no-really) v2 with VS2010 support

It quickly became apparent that getting VS2010 support now was more important than getting the new shiny v2 bits, so without further ado protobuf-net (v1) with VS2010 support is now available on the project site.

As I say, building VS packages isn’t exactly something you want to do often (unless you do it routinely enough to have a highly polished and fully automated process), and I wasn’t 100% sure that the VS9 SDK API would work against VS10. Fortunately, it all went smoothly. Too smoothly! If you have your own custom packages, you may well find that simply duplicating your 9.0 hive changes into the 10.0 key does the job, giving you an installer that targets both IDEs.

On a v2 note; this really is getting there now; most of the core logic is implemented (although there is still a fair bit of testing to do), and it is looking very promising. I hope to be able to release something over the next few weeks.

You might guess (correctly) that I’m a bit peeved about the Apple shenanigans – but whichever way that plays out I’m content: protobuf-net v2 is quite simply a superior product to v1, and I’ve had lots of geek-fun (which is like regular fun, but harder to explain) learning the guts of IL; previously largely a read-only topic to me.

Any problems / feedback – please let me know.