Introducing the protobuf-net precompiler

Over the last few posts (here and here) I’ve given a few hints as to compiling for other platforms. Basically, this all relates to how well protobuf-net works on platforms like iOS, WinRT/Metro, Silverlight, Phone 7, etc. These heavily restricted runtimes don’t allow much meta-programming, and they might be running on low-power CPUs, so reflection (even if possible) is not ideal.

I’ve played with assembly generation before, but with mixed results. For example, here for Phone 7. This could just about work for some frameworks, but was patchy on some, and won’t work at all for others.

Well, all that IKVM shininess has opened up a whole new set of tools. The small beauty that I’m ridiculously pleased with is a new utility exe in the SVN trunk (will be part of a proper release soon): precompile.

This unassuming little tool works a bit like “sgen”, but with the ability to target multiple frameworks. What it does is:

• inspect the input assembly (or assemblies) to resolve (if it can) the target framework
• initialize an IKVM universe targeting that framework
• load the core framework libraries, protobuf-net, and the input assemblies into the IKVM universe
• scan the input assemblies for types marked [ProtoContract]
• add those to a protobuf-net model
• compile the model to a type / assembly of your choosing

To use that in a project you might:

• create a new DTO project in your chosen framework, and compile it
• execute precompile to generate to a serializer assembly
• from your application project, reference the DTO and serializer assemblies
• use the type you created

For example, say I create a new Phone 7 DTO assembly, called PhoneDto (because it is late and I lack imagination). I can then create a serialization assembly via:

precompile {some path}\PhoneDto.dll –o:PhoneSerializer.dll –t:MySerializer

This will generate a library called PhoneSerializer.dll, which you can reference from your main project (in addition to the DTO and the protobuf-net core).

Then, just use MySerializer:

var ser = new MySerializer();
ser.Serialize(output, obj);

I hope this finally solves a number of tooling issues. I’m pretty pleased with it. I’ve tested it against a range of different frameworks, and it has worked well – but if you get problems, just let me know (comment here, or email me, or log an issue on the protobuf-net project site).

Enter the IKVM

aka Meta-programming and Metro for .NET

In my previous blog entry I gave an overview of how protobuf-net is arranged internally, and hinted that it all falls apart for Metro. So: what goes wrong? Currently, protobuf-net has a lot (and I do mean a lot) of code built on top of ILGenerator, the reflection class that underpins meta-programming. This class is great for hardcore library builders: you can build individual methods (via DynamicMethod) or entire assemblies (via AssemblyBuilder). It is very low level, since you are writing raw IL – but I’m not completely crazy, so I wrap that up in some utility methods. There are a few problem., though:

• ILGenerator usually expects be be working on the current framework
• Even though you can use a “reflection only load”, this still fails horribly for System.Runtime.dll (aka Metro for .NET)
• In early versions of .NET (and I try to support .NET 2 and upwards), the ability to properly inspect attribute data against reflection-only types/members (GetCustomAttributesData) is simply missing

So… basically, it all falls to pieces for generating as assembly targeting Metro for .NET, based on a Metro for .NET input assembly.

What I would need is some kind of utility that is:

• Broadly similar to Reflection.Emit, so I don’t have to rewrite a few thousand lines of code (since I want to keep that code for the runtime meta-programming on full .NET)
• Able to load and work with alternative frameworks without getting confused
• Able to give full attribute information, even on .NET 2
• Inbuilt or free, to match the existing license

Who would go to the trouble of writing such a thing?

Maybe somebody who is already writing compiler-like tools that aren’t tied to a specific framework, and who has run into the existing limitations of Reflection.Emit? Maybe somebody writing a Java/.NET bridge? Like IKVM.NET ?

Actually, for my purposes I don’t need most of IKVM. I just need one tiny piece of it: IKVM.Reflection. This library was specifically written to mimic the existing Reflection.Emit API, but without all the problems. In particular, it has a Universe class that is a bit like an AppDomain; you load assemblies into a Universe, and that is what is used for resolution etc after that. Here’s a simple example – which will immediately make sense to anyone used to Reflection.Emit – or another example specifically targeting Metro for .NET. The nice thing about the API is that mostly I can change between IKVM and Reflection.Emit via a simple:

#if FEAT_IKVM
using Type = IKVM.Reflection.Type;
using IKVM.Reflection;
#else
using System.Reflection;
#endif

OK, I would be grossly exaggerating if I claimed that was the only change I had to make, but that’s the most demonstrable bit. What this means is that just for my cross-platform compiler, everything in the Model, Strategy and Compiler modules (see previous blog) switches to IKVM terminology. The Core still uses System terminology, and the Runtime doesn’t apply (I obviously can’t instantiate/execute types/methods from another framework, even if I can inspect them).

With this technique, I can now successfully load a Metro for .NET assembly (such as a DTO), and generate a fully static-compiled assembly (“Standalone” on the diagram) targeting Metro for .NET. No reflection at runtime, no nasty hacks – just: an assembly that wasn’t generated by the MS tools.

I’m still working on turning this into a slicker tool (comparable to, say, SGEN), but a working illustration is in the repo; in particular, see MetroDTO (some sample DTOs), TestIkvm (my proof-of-concept compiler hard-coded to MetroDTO), and Metro_DevRig (which shows the generated assembly working in a Metro for .NET application, including performance comparisons to XmlSerializer and DataContractSerializer).

Additionally, it seems extremely likely that the same tool should be able to write platform-targeted assemblies for Silverlight, Phone 7, XNA, CF, etc. Which is nice.

Thanks and Acknowledgements

I’m hugely indebted to Jeroen Frijters in all of this; both for providing IKVM.Reflection, and for his direct and very prompt assistance when playing with all the above. Mainly for correcting my own brain-dead mistakes, but also for a very quick bug-fix when needed. The library is awesome, thanks.

(as a small caveat, note that protobuf-net is currently exposing a locally built version of IKVM.Reflection; this will be rectified after the next dev build of IKVM)

Some internals of protobuf-net

This post is a bit of background explanation for my next, much more interesting (IMO) post. But in order for that one to make sense without being too long, I wanted to talk about the “what?” and “why?” first, and then discuss the “how?” after. Make sense?

First, I’ll whet your appetite with this:

This is protobuf-net running nice and fast (much faster than XmlSerializer or DataContractSerializer) on .NET for Metro style apps. Which is a pretty neat trick, because Metro doesn’t allow any of the usual tricks that protobuf-net uses.

So what does it use?

A walkthrough of protobuf-net

The first piece of protobuf-net is the core – this is basically the fundamental reader/writer API (think: XmlReader/XmlWriter), and some utility methods. Every build of protobuf-net includes the core, although many parts may have different implementations on a per-platform basis.

Next we have the model; this is where the user defines the types that are going to be serialized by the model. In many cases, the model is populated automatically as you serialize items (it figures out what is needed). The model doesn’t know how to do anything – it only knows the configuration.

Next comes the strategy; a series of decorators and other structures that turn that configuration into something resembling a sensible way to handle, each configuration, building from simpler steps.

And then given a strategy, we need some mechanism to implement that strategy. A simple option is reflection, which is convenient and available on most platforms, but comparatively slow.

A much more elegant approach is to use meta-programming to turn a strategy into regular IL code. This is pretty complex, but leads to very fast results. Unfortunately, a lot of light-weight platforms do not allow this.

Which is a shame, because once you have a working compiler, you can go so far as to generate (via a utility exe at build time) a standalone serialization assembly, which then doesn’t need to do any thinking at runtime.

In fact, if you do this you don’t even need anything else – just the core and a serialization assembly. This technique allows you to generate, with a little voodoo, a pair of small assemblies that can be used on platforms such as MonoTouch, as described here.

So you just did that, right? Doesn’t sound too bad…

You’d think so, wouldn’t you? Unfortunately, there’s a stumbling block. Generating a standalone assembly is … complex. It only works on full .NET for starters, but also: Reflection.Emit only works with full .NET assemblies. You can’t load your Metro DTO into regular .NET, spin up a TypeBuilder, and run with it; a: you won’t be able to load the types correctly (since your Metro DTO targets an entirely different framework), and b: what you generate won’t be convincing as a Metro dll. We get away with it for MonoTouch, but that is just because the Xamarin folks are evil geniuses.

What we need, then, is some way of generating a Metro serialization dll, after inspecting a Metro DTO dll, but presumably as a build tool running on regular .NET. That, as they say, gets tricky. And that is for next time.