Introducing the protobuf-net precompiler

Over the last few posts (here and here) I’ve given a few hints as to compiling for other platforms. Basically, this all relates to how well protobuf-net works on platforms like iOS, WinRT/Metro, Silverlight, Phone 7, etc. These heavily restricted runtimes don’t allow much meta-programming, and they might be running on low-power CPUs, so reflection (even if possible) is not ideal.

I’ve played with assembly generation before, but with mixed results. For example, here for Phone 7. This could just about work for some frameworks, but was patchy on some, and won’t work at all for others.

Well, all that IKVM shininess has opened up a whole new set of tools. The small beauty that I’m ridiculously pleased with is a new utility exe in the SVN trunk (will be part of a proper release soon): precompile.

This unassuming little tool works a bit like “sgen”, but with the ability to target multiple frameworks. What it does is:

• inspect the input assembly (or assemblies) to resolve (if it can) the target framework
• initialize an IKVM universe targeting that framework
• load the core framework libraries, protobuf-net, and the input assemblies into the IKVM universe
• scan the input assemblies for types marked [ProtoContract]
• add those to a protobuf-net model
• compile the model to a type / assembly of your choosing

To use that in a project you might:

• create a new DTO project in your chosen framework, and compile it
• execute precompile to generate to a serializer assembly
• from your application project, reference the DTO and serializer assemblies
• use the type you created

For example, say I create a new Phone 7 DTO assembly, called PhoneDto (because it is late and I lack imagination). I can then create a serialization assembly via:

precompile {some path}\PhoneDto.dll –o:PhoneSerializer.dll –t:MySerializer

This will generate a library called PhoneSerializer.dll, which you can reference from your main project (in addition to the DTO and the protobuf-net core).

Then, just use MySerializer:

var ser = new MySerializer();
ser.Serialize(output, obj);

I hope this finally solves a number of tooling issues. I’m pretty pleased with it. I’ve tested it against a range of different frameworks, and it has worked well – but if you get problems, just let me know (comment here, or email me, or log an issue on the protobuf-net project site).

Enter the IKVM

aka Meta-programming and Metro for .NET

In my previous blog entry I gave an overview of how protobuf-net is arranged internally, and hinted that it all falls apart for Metro. So: what goes wrong? Currently, protobuf-net has a lot (and I do mean a lot) of code built on top of ILGenerator, the reflection class that underpins meta-programming. This class is great for hardcore library builders: you can build individual methods (via DynamicMethod) or entire assemblies (via AssemblyBuilder). It is very low level, since you are writing raw IL – but I’m not completely crazy, so I wrap that up in some utility methods. There are a few problem., though:

• ILGenerator usually expects be be working on the current framework
• Even though you can use a “reflection only load”, this still fails horribly for System.Runtime.dll (aka Metro for .NET)
• In early versions of .NET (and I try to support .NET 2 and upwards), the ability to properly inspect attribute data against reflection-only types/members (GetCustomAttributesData) is simply missing

So… basically, it all falls to pieces for generating as assembly targeting Metro for .NET, based on a Metro for .NET input assembly.

What I would need is some kind of utility that is:

• Broadly similar to Reflection.Emit, so I don’t have to rewrite a few thousand lines of code (since I want to keep that code for the runtime meta-programming on full .NET)
• Able to load and work with alternative frameworks without getting confused
• Able to give full attribute information, even on .NET 2
• Inbuilt or free, to match the existing license

Who would go to the trouble of writing such a thing?

Maybe somebody who is already writing compiler-like tools that aren’t tied to a specific framework, and who has run into the existing limitations of Reflection.Emit? Maybe somebody writing a Java/.NET bridge? Like IKVM.NET ?

Actually, for my purposes I don’t need most of IKVM. I just need one tiny piece of it: IKVM.Reflection. This library was specifically written to mimic the existing Reflection.Emit API, but without all the problems. In particular, it has a Universe class that is a bit like an AppDomain; you load assemblies into a Universe, and that is what is used for resolution etc after that. Here’s a simple example – which will immediately make sense to anyone used to Reflection.Emit – or another example specifically targeting Metro for .NET. The nice thing about the API is that mostly I can change between IKVM and Reflection.Emit via a simple:

#if FEAT_IKVM
using Type = IKVM.Reflection.Type;
using IKVM.Reflection;
#else
using System.Reflection;
#endif

OK, I would be grossly exaggerating if I claimed that was the only change I had to make, but that’s the most demonstrable bit. What this means is that just for my cross-platform compiler, everything in the Model, Strategy and Compiler modules (see previous blog) switches to IKVM terminology. The Core still uses System terminology, and the Runtime doesn’t apply (I obviously can’t instantiate/execute types/methods from another framework, even if I can inspect them).

With this technique, I can now successfully load a Metro for .NET assembly (such as a DTO), and generate a fully static-compiled assembly (“Standalone” on the diagram) targeting Metro for .NET. No reflection at runtime, no nasty hacks – just: an assembly that wasn’t generated by the MS tools.

I’m still working on turning this into a slicker tool (comparable to, say, SGEN), but a working illustration is in the repo; in particular, see MetroDTO (some sample DTOs), TestIkvm (my proof-of-concept compiler hard-coded to MetroDTO), and Metro_DevRig (which shows the generated assembly working in a Metro for .NET application, including performance comparisons to XmlSerializer and DataContractSerializer).

Additionally, it seems extremely likely that the same tool should be able to write platform-targeted assemblies for Silverlight, Phone 7, XNA, CF, etc. Which is nice.

Thanks and Acknowledgements

I’m hugely indebted to Jeroen Frijters in all of this; both for providing IKVM.Reflection, and for his direct and very prompt assistance when playing with all the above. Mainly for correcting my own brain-dead mistakes, but also for a very quick bug-fix when needed. The library is awesome, thanks.

(as a small caveat, note that protobuf-net is currently exposing a locally built version of IKVM.Reflection; this will be rectified after the next dev build of IKVM)

Some internals of protobuf-net

This post is a bit of background explanation for my next, much more interesting (IMO) post. But in order for that one to make sense without being too long, I wanted to talk about the “what?” and “why?” first, and then discuss the “how?” after. Make sense?

First, I’ll whet your appetite with this:

This is protobuf-net running nice and fast (much faster than XmlSerializer or DataContractSerializer) on .NET for Metro style apps. Which is a pretty neat trick, because Metro doesn’t allow any of the usual tricks that protobuf-net uses.

So what does it use?

A walkthrough of protobuf-net

The first piece of protobuf-net is the core – this is basically the fundamental reader/writer API (think: XmlReader/XmlWriter), and some utility methods. Every build of protobuf-net includes the core, although many parts may have different implementations on a per-platform basis.

Next we have the model; this is where the user defines the types that are going to be serialized by the model. In many cases, the model is populated automatically as you serialize items (it figures out what is needed). The model doesn’t know how to do anything – it only knows the configuration.

Next comes the strategy; a series of decorators and other structures that turn that configuration into something resembling a sensible way to handle, each configuration, building from simpler steps.

And then given a strategy, we need some mechanism to implement that strategy. A simple option is reflection, which is convenient and available on most platforms, but comparatively slow.

A much more elegant approach is to use meta-programming to turn a strategy into regular IL code. This is pretty complex, but leads to very fast results. Unfortunately, a lot of light-weight platforms do not allow this.

Which is a shame, because once you have a working compiler, you can go so far as to generate (via a utility exe at build time) a standalone serialization assembly, which then doesn’t need to do any thinking at runtime.

In fact, if you do this you don’t even need anything else – just the core and a serialization assembly. This technique allows you to generate, with a little voodoo, a pair of small assemblies that can be used on platforms such as MonoTouch, as described here.

So you just did that, right? Doesn’t sound too bad…

You’d think so, wouldn’t you? Unfortunately, there’s a stumbling block. Generating a standalone assembly is … complex. It only works on full .NET for starters, but also: Reflection.Emit only works with full .NET assemblies. You can’t load your Metro DTO into regular .NET, spin up a TypeBuilder, and run with it; a: you won’t be able to load the types correctly (since your Metro DTO targets an entirely different framework), and b: what you generate won’t be convincing as a Metro dll. We get away with it for MonoTouch, but that is just because the Xamarin folks are evil geniuses.

What we need, then, is some way of generating a Metro serialization dll, after inspecting a Metro DTO dll, but presumably as a build tool running on regular .NET. That, as they say, gets tricky. And that is for next time.

The trouble with tuples

A little while ago, I mentioned how new tuple handling in protobuf-net meant that likely-looking tuples could now be handled automatically, and even better this meant I could remove my hacky KeyValuePair<,> handling. All was well in the world – after all, what could go wrong?

Answer: Mono.

Actually, I’m not criticizing the Mono implementation really, but simply: Mono has a subtly different implementation of KeyValuePair<,> to Microsoft – nothing huge; simply there exist some set accessors (private ones, visible here). And the library was pretty fussy – if there was a set it wouldn’t be treated as a sure-thing tuple.

This is now fixed in r447, but: if you are using Mono and your dictionaries stopped serializing – then honestly and humbly: sorry about that.

And if you are a library author – watch out: sometimes the simplest most subtle differences between implementations can kill you.

Automatic serialization; what’s in a tuple?

I recently had a lot of reason (read: a serialization snafu) to think about tuples in the context of serialization. Historically, protobuf-net has focused on mutable types, which are convenient to manipulate while processing a protobuf stream (setting fields/properties on a per-field basis). However, if you think about it, tuples have an implicit obvious positional “contract”, so it makes a lot of sense to serialize them automatically. If I write:

var tuple = Tuple.Create(123, "abc");

then it doesn’t take a lot of initiative to think of tuple as an implicit contract with two fields:

• field 1 is an integer with value 123
• field 2 is a string with value “abc”

Since it is deeply immutable, at the moment we would need to either abuse reflection to mutate private fields, or write a mutable surrogate type for serialization, with conversion operators, and tell protobuf-net about the surrogate. Wouldn’t it be nice if protobuf-net could make this leap for us?

Well, after a Sunday-night hack it now (in the source code) does.

The rules are:

• it must not already be marked as an explicit contract
• only public fields / properties are considered
• any public fields (spit) must be readonly
• any public properties must have a get but not a set (on the public API, at least)
• there must be exactly one interesting constructor, with parameters that are a case-insensitive match for each field/property in some order (i.e. there must be an obvious 1:1 mapping between members and constructor parameter names)

If all of the above conditions are met then it is now capable of behaving as you might hope and expect, deducing the contract and using the chosen constructor to rehydrate the objects. Which is nice! As a few side-benefits:

• this completely removes the need for the existing KeyValuePairSurrogate<,>, which conveniently meets all of the above requirements
• it also works for C# anonymous types if we want, since they too have an implicit positional contract (I am not convinced this is significant, but it may have uses)

This should make it into the next deploy, once I’m sure there are no obvious failures in my assumptions.

Just one more thing, sir…

While I’m on the subject of serialization (which, to be fair, I often am) – I have now also completed some changes to use RuntimeHelpers.GetHashCode()for reference-tracking (serialization). This lets met construct a reference-preserving hash-based lookup to minimise the cost of checking whether an object has already been seen (and if so, fetch the existing token). Wins all round.

So soon, beta 2

I’m delighted by the level of response to my protobuf-net v2 beta; I have been kept really busy with a range of questions, feature requests and bug fixes (almost exclusively limited to the new features).

So, before I annoy people with too many “fixed in source” comments, I thought I’d better re-deploy. Beta 2 gives the same as beta 1, but with:

• reference-tracked objects
  • root object now included (BREAKING CHANGE)
  • fixed false-positive recursion issue
• full type metadata
  • support custom formatting/parsing
• interface serialization
  • support serialization from interfaces
  • support known-implementations of interfaces
  • support default concrete type
• WCF
  • expose WCF types on public API to allow custom models
• misc
  • support shadow set-methods
  • fix IL glitch
  • fix cold-start thread-race condition
  • fix missing method-forwarding from legacy non-generic API

In addition to a very encouraging level of interest, I’m also pleased that things like the interface-based support mentioned above were pretty effortless to push through the v2 type-model. These are things that had been proposed previously, but there was just no way to push them into the codebase without hideous (and cumulative) hackery. With v2, it was pretty much a breeze.

But for v2 fun see the project download page

And please, keep pestering me ;p

Note that the BREAKING CHANGE marked above only applies to data serialized with the new full-graph AsReference option. No v1 data is impacted, but if you have stored data from earlier v2 alpha/beta code, please drop me a line and I can advise.

protobuf-net v2, beta

It has been a long time in the coming, I know. I make no excuses, but various other things have meant that it didn’t get much input for a while…

But! I’m happy to say that I’ve just pushed a beta download up onto the project site

So… this v2… what is it?

Basically, it is still the core protobuf stream, but against all the safe advice of my employer I rewrote the entire core. For many reasons:

• the original design grew organically, and ideas that seemed good turned to make for some problematic code later
• the vast overuse of generics (and in particular, generics at runtime via reflection) was really hurting some platforms
• there were many things the design could not readily support – structs, runtime models, pre-built serialization dlls, usage on mobile devices, etc
• the code path at runtime was just not as minimal as I would like
• and a myriad of other things

So… I rewrote it. Fortunately I had a barrage of integration tests, which grew yet further during this exercise. The way I figured, I had 2 choices here:

• go with a richer code-generator to use as a build task (not runtime)
• go crazy with meta-programming

After some thought, I opted for the latter. In particular, this would (in my reasoning) give me lots of flexibility for keeping things runtime based (where you are able to, at least), and would be a good opportunity to have some fun and learn IL emit to dangerous levels (which, incidentally, turns out to be very valuable – hence mine and Sam’s work on dapper-dot-net).

So what is new?

I don’t claim this list is exhaustive:

• the ability to define models at runtime (also useful for working with types you don’t control) (example)
• the ability to have separate models for the same types
• pre-generation to dll
• runtime performance improvements
• support for structs
• support for immutable objects via surrogates (example)
• the ability to run on mobile platforms (unity, wp7 – and perhaps moot for a while: MonoDroid, MonoTouch – but presumably Xamarin)
• the ability to serialize object graphs preserving references (example)
• the ability to work with types not known in advance (same)
• probably about 40 other things that slip my mind

I’ll try to go into each of these in detail when I can

And what isn’t there yet?

This is beta; at some point I had to make a milestone cut – but some things aren’t quite complete yet; these are not usually needed (hint: I’m happy to use protobuf-net v2 on my employer’s code-base, so I have confidence in it, and confidence that the core paths have been smoke-tested)

But things that definitely aren’t there in the beta

• WCF hooks
• VS tooling (it is just the core runtime dlls)
• round-trip safe / “extension” fields
• extraction of .proto (schema) files from a model
• my build / deploy script needs an overhaul
• tons of documentation / examples
• tooling to make the “mobile” story easier (but; it should work – it is being used in a few, for example)

But; have at it

If you want to play with it, go ahead. I only ask that if something behaves unexpectedly, drop me a line before declaring loudly “it sux, dude!”. It might just need some guidance, or maybe even a code fix (I’m far from perfect).

Likewise, any feature suggestions, etc; let me know.

DataTable – life in the old beast?

Updated: Billy referred me to the binary remoting format for DataTable; stats for BinaryFormatter now include both xml and binary.

---

This week, someone asked the fastest way to send a lot of data, only currently available in a DataTable (as xml), over the wire.

My immediate thought was simply GZIP the xml and have done with it, but I was intrigued… can we do better? In particular (just for a complete surprise) I was thinking whether protobuf-net could be used to write them. Now don’t get me wrong; I’m not a big supporter of DataTable, but they are still alive in the wild. And protobuf-net is a general-purpose serializer, so… why not try to be helpful, eh?

v1 won’t help at all with this, since the object-model just isn’t primed for extension, but in v2 we have many options. I thought I’d look at some typical data (SalesOrderDetail from AdventureWorks2008R2; 121317 rows and 11 columns), comparing the inbuilt xml support, BinaryFormatter support, and protobuf-net. And each of those uncompressed, via gzip, and via deflate.

At this point my new friend Lucy (right) is pestering me for a walk, but… it worked; purely experimental, but committed (r356). Results shown below:

Table loaded    11 cols 121317 rows
DataTable (xml) (vanilla)               2269ms/6039ms
                                       64,150,771 bytes
DataTable (xml) (gzip)                  4881ms/6714ms
                                       7,136,821 bytes
DataTable (xml) (deflate)               4475ms/6351ms
                                       7,136,803 bytes
BinaryFormatter (rf:xml) (vanilla)      3710ms/6623ms
                                       74,969,470 bytes
BinaryFormatter (rf:xml) (gzip)         6879ms/8312ms
                                       12,495,791 bytes
BinaryFormatter (rf:xml) (deflate)      5979ms/7472ms
                                       12,495,773 bytes
BinaryFormatter (rf:binary) (vanilla)   2006ms/3366ms
                                       11,272,592 bytes
BinaryFormatter (rf:binary) (gzip)      3332ms/4267ms
                                       8,265,057 bytes
BinaryFormatter (rf:binary) (deflate)   3216ms/4130ms
                                       8,265,039 bytes
protobuf-net v2 (vanilla)               316ms/773ms
                                       8,708,846 bytes
protobuf-net v2 (gzip)                  932ms/1096ms
                                       4,797,856 bytes
protobuf-net v2 (deflate)               872ms/1050ms
                                       4,797,838 bytes

Since “walkies” beckons, I’ll be brief:

• the inbuilt xml and BinaryFormatter (as xml) are both very large natively and compress pretty well, but with noticeable additional CPU costs
• with BinaryFormatter using the binary encoding it does much less work, getting (before compression) into the same size-bracket as the xml encoding managed with compression
• protobuf-net is much faster in terms of CPU, but slightly larger than the gzip xml output
• unusually for protobuf, it compresses quite will – presumably there is enough text in there to make it worthwhile; this then takes roughly half the size and much less CPU (compared to gzip xml)

Overall the current experimental code is exceptionally rough and ready, mainly as an investigation. But… should I pursue this? do people still care about DataTable enough? Or just use gzip/xml in this scenario?

Walkthrough: protobuf-net on Phone 7

UPDATE: better tools for this now exist; see precompile

Now’s my chance to demonstrate the (pre-alpha) “v2” protobuf-net API, while at the same time showing it working on Phone 7. In particular, this shows the new pre-compilation functionality, which is vital for good performance on Phone 7, iPhone, XNA, Compact Framework, etc (since they have scant meta-programming).

But first…

Why? Just why?

You have a mobile device; data should be considered a limited resource, both in terms of financial cost (data plans etc) and bandwidth-related performance. Therefore we want to send less over the wire.

• Google’s protobuf wire-format is extremely dense. In theory you can get a few bytes tighter if you squeeze every nibble manually, but you’ll go crazy in the process.
• Mobile devices rarely have full meta-programming; we don’t want reflection (too slow), so we want fully pre-compiled serializers.
• Code-generation is one option, but most code-gen APIs want you to use their object model. Thanks very much, but I already have an object model. I’ll use that, ta.
• Shiny.

Building your domain model

You can use protobuf-net from a .proto (Google’s DSL for describing protobuf schemas), but it is my belief that most .NET developers want to work with their existing model. So let’s start with a basic model.

For this demo, I’m starting with a Silverlight library project to represent my model (or it could also be a DTO layer). There are various ways of setting this up, but this is the simplest while I work out the kinks…

My model is very basic; just an order header / detail pair. For simplicity I’m using the protobuf-net attributes to allocate field-numbers to the properties, but there are may other ways of doing this now (including support for vanilla objects with no adornments). Here we go; very simple:

Generating a serialization assembly

This is the most obvious difference in “v2”; you can now pre-generate the serialization code into a fully static-typed assembly. The “generate at runtime” model is still supported for “regular” .NET, and has also had a complete overhaul improving performance noticeably.

“The plan” here is to write a simply utility exe that you can call in your build process to do this step, but I don’t have that complete yet. Instead, just for this demo, I’m going to use “regular” .NET console exe, referencing the “regular” protobuf-net assembly (and the DTO assembly), with an entire 4 lines of code:

This tells the API what types we are interested in, and offers a wide range of options for changing how the model is mapped. We’re passing in “true” to let it figure out the details itself.

Running this exe generates “MySerializer.dll”, which is our serialization assembly with a class “OrderSerializer”. It still needs some utility code from protobuf-net, so in our Phone 7 app we’ll need references to the DTO assembly, the serializer assembly and the Phone 7 version of protobuf-net.

 

Now all we need to do is use the serializer! In “v1” we would have used static methods on “ProtoBuf.Serializer”, but instead we want to use our specific pre-generated type:

We also need to create some sample data to play with:

Next, purely for illustration we’ll ignore the handy “DeepClone” method, and serialize / deserialize the object via a MemoryStream:

And because I have extremely limited UI skills, I’ll just throw the cloned data into a ListBox:

Et voila; working protobuf-net serialization via a pre-generated serialization dll, on Phone 7.

Initial performance tests seem to indicate that it is in the region of 30-50 times faster than the DataContractSerializer that ships with Phone 7, and the protobuf wire format typically takes about 20% of the bandwidth. Win win.

Caveats

The “v2” code isn’t complete yet! I’m still working through all the compatibility tests. It is getting there, though. It is a week or so away from a stable beta, I suspect.

Additionally, I’m still working through some of the issues associated with using “regular” .NET to generate the assembly for a light framework such as CF / Phone 7. Some combinations seem to work better than others.

Binary data and strings

You have no idea how often I get an e-mail that talks about binary data (for me, usually protobuf data) and string encoding (usually via utf-8).

Here’s the thing folks: most binary data cannot be translated to a string with a utf-* translation. It just isn’t possible. If you are talking about an arbitrary byte sequence that you need to store or transmit as a string, then your best bet is base-64.

Fortunately, in .NET Convert.ToBase64String and Convert.FromBase64String deal with all the details, making it trivial to use.

I keep talking myself into (and then back out-of) adding string-based overloads (etc) to protobuf-net; maybe I’ll add them just to stop the internal debate…

protobuf-net v2 on the iPhone

It might not look very impressive to you, but this is pure joy to me. It is the first evidence I’ve seen that the revised rewritten protobuf-net can work in MonoTouch for use on the iPhone – and presumably MonoDroid in the future.

A big thanks here to Christian Weyer and Andrew Rimmer who both offered some help here (since I don’t actually own a mac, an iPhone or MonoTouch!). The grab is from Andrew (he assures me it works fine on the actual device too).

This piece of work has been a slog, and I kept putting it off (because frankly, re-writing the entire stack is a huge job). The goals have not been trivial:

• Continue to support all the existing runtime-only usage…
• ..while also making it support fully static / AOT compatible
• Allow it to write serialization dlls in advance (like sgen)
• Fix the stability issues on Compact Framework (caused by overuse of generics)
• Remove some performance bottlenecks on Compact Framework
• Support POCO (or more accurately, work without requiring special [attributes]) so you can use it with types outside your influence, or for inheritance that isn’t known at compile-time
• Make it work on iPhone etc
• Make it even faster
• Make it more extensible (for example, what if I want to write a proxy that writes a DataTable to a protobuf stream?)
• Make it support mutable structs (yes I know that they are usually evil, but people simply keep asking for this, in particular for XNA)
• Remove a heap of code-debt caused by an evolving project that grew without a 100% clear direction
• Throw in some other feature-requests and bug-fixes for good measure
• And as a side-benefit, it even compiles and runs on .NET 1.1 or via mcs on Mono (with minimal #if usage)

I should stress that it is not yet complete – but the core of the engine is there and working, demonstrating most of the key points above (albeit in a very raw state). There’s still a way to go, but it is getting there.

Distributed caching with protobuf-net

I’ve been playing with distributed caches lately, and of course this is a really good use-case for something like protobuf-net – if there is anywhere you want to minimise your memory costs, network IO costs and CPU costs, it is your cache (it gets kinda busy).

As a pure example of “which system can I get installed and running the quickest”, I looked at memcached, using the windows service from here and the enyim.com client. Not only were they a breeze to get working, but this setup seems pretty popular, and also covers memcached providers, which uses enyim under the hood.

Fortunately, the enyim client has an extension-point that allows you to nominate a serializer through your config file (actually it calls it a transcoder, but I’m not sure it fits the definition). This allows us to swap BinaryFormatter for protobuf-net, which (for suitable types) has huge savings – in my crude tests I was seeing only 20% of the original network traffic, and roughly 75% of the original CPU costs.

Enough bluster…

OK, so in r282 I’ve added just such a transcoder (I still don’t like the name…) – and the good thing it is so effortless to configure it:

Before:

<enyim.com>
   <memcached>
       <servers>
           <add address="127.0.0.1" port="11211" />

After:

<enyim.com>
   <memcached transcoder="ProtoBuf.Caching.Enyim.NetTranscoder, protobuf-net.Extensions">
       <servers>
           <add address="127.0.0.1" port="11211" />

Wasn't that was painless? and some huge benefits (although obviously you’d want to run some integration/regression tests if applying this to an existing system). Oh, and of course you need to include protobuf-net and protobuf-net.Extensions.dll in your project, in addition to the enyim dll.

My only niggle is that because there is no way of knowing the type up-front, I had to include the outermost type information on the wire (hence the “Net” in the transcoder’s name, since it won’t be truly portable). I could get around this by having the caller indicate the type, but the enyim dll doesn’t play that game.

What next?

If you find it good/bad/pointless/buggy please let me know.

If you’ve got another cache that might benefit from some protobuf-net love, please let me know. Velocity, perhaps? Azure?

State Assassins with experience of Reporting Services

No, this isn’t one of the more unlikely job listings on careers – instead, a bit of work from last week that I found pretty interesting.

I was essentially adding some consultancy input to an unfamiliar codebase; the problem was that the site would be running very smoothly, then page-times would go crazy for a while, and then suddenly go back to normal.

It turned out (after putting state into SQL-Server for analysis) that some of the state records were unexpectedly large. Damagingly huge. So what were they? After all, the app only used state to store one tiny innocent object…

One advantage of using a database for state is that you can get the data back out, and it turns out that it isn’t too hard to reverse a BLOB back into the state objects (it helped that stackoverflow had half the answer already).

The cuplrit? You might guess (from the post title): reporting services and the web-forms report viewer control. As soon as a user viewed a report, it was using state to cache the data – which might work great for small reports, but if the report is large it gets… fun.

Perhaps a design failing in ASP.NET state; once a large object is in state, it goes up and down the wire (between the web-server and the state storage) on every state-enabled request for that user, even when no longer using that data. So if they used the report 10 minutes ago, and are now just looking at nice slim tidy pages, they are still dragging a huge state object over the network all the time, potentially crippling the site for lots of other users.

If the page had been highly AJAX-enabled it could have been much worse. And the reason the problem ended so abruptly? My reasoning is that the user in question (who is harming the site through no fault of their own) got bored of sluggish responses and did something else for a while.

Learning points:

• Even if you aren’t using SQL Server for state normally, it makes analysis much easier as a temporary swap-out
• Watch out for web-controls (especially reports) that might (ab)use state
• You can decipher a regular state BLOB – it isn’t tricky (I’ll update that SO question with an example if anyone is interested)
• Consider partitioning your app, or disabling state in some scenarios
• We aware of how much data your reports are dragging around with them

Generics, and When Not To

Generics are undoubtedly of huge benefit for writing maintainable, static typed code with the minimum of fuss... but can it be overused? I firmly believe that generics are great for regular line-of-business code, with typed collections/lists and everything else that goes with them. But what about utility code, in particular when you're doing something complex with reflection... maybe give it a miss.

A cautionary tale: if you find yourself doing something with generics that looks complex... stop; you might be using the wrong approach. Of course, you might be right after all - but at least do a sanity check.

Consider protobuf-net; this is a member serializer (akin to XmlSerializer, DataContractSerializer, et al - but writing binary). It is quite a complex setup, but essentially it is using a strategy pattern (via reflection) to serialize each of the different properties. To avoid lots of boxing/casting, and to allow me to use the typed form of delegate invoke (much faster than dynamic invoke), it started out with a highly generic model - i.e. strategy classes that use generics to describe the input, output and anything else required.

But the problem with generics in this context is that they beget more generics... you start with something simple like "Foo<T>", and before you know it you've got types with multiple generic type arguments and constraints, inheriting from other generic types, all the way down (just like the turtles). Understanding the type signature alone is headache inducing - never mind trying to use reflection (MakeGenericType etc).

More importantly, however - the generic approach doesn't work very well if you want to use a decorator-based strategy; for example, to serialize the property "public List<Foo> {get;}", we might have a decorator that does property access, a decorator that handles list iteration, and a decorator that serializes Foo (and whatever that involves). You simply can't write that just using generics, as you can't tell in advance how deep the hole goes, and how many type arguments you'll need (let alone what they might be).

For utility code (such as a serialization engine) there are some other considerations:

• Some frameworks (Compact Framework) have limitations in the number of generic types you can create at runtime
• Some frameworks (Micro Framework and .NET 1.1) simply don't have generics at all

So faced with ever-increasing code complexity (always a bad sign), and it not working reliably (or at all) on some frameworks, I've decided to try to drop generics completely from the core implementation, and just have an "object" based decorator-chain. It then doesn't matter how complex the chain gets - I know I can still pass an object downstream (even if I've changed the type), and it won't offend CF/MF.

So what about performance? I certainly don't want to cripple the code...

• Passing objects between the different decorators in a chain might look ungainly, but actually the odd bit of boxing and casting isn't necessarily a major problem (especially if it means the code works)
• Where available (i.e. framework dependent), we can still use Delegate.CreateDelegate and the generic wrapper trick, but now as an implementation detail of an otherwise non-generic decorator (i.e. the public API would still be non-generic)
• Where available (i.e. framework dependent), we have the option of making our decorators support Reflection.Emit - so rather than having methods that do the operation, we can have methods that write how to do it. There are several performance advantages to this:
  • no boxing/unboxing/casting
  • no stack hops
  • no delegate invoke (dynamic or typed)

The refactor is still incomplete, but is slowly taking shape (although it is taking longer than I hoped). It helps that I have a good set of unit tests so that I know just how far I have to go... but I've done enough to satisfy me that a: it should work much more reliably on CF, b: it should still fly on the regular framework, and c: if I really want I could probably make it work on .NET 1.1 and MF (but I'm not promising to do this).

Obfuscation, serialization and automatically implemented properties

Not the first time, I saw a question today discussing obfuscation and serialization in .NET; as a re-cap, obfuscation (in this context) is the art of making an assembly hard to read, and one of the most basic tricks is to rename types and members to things like "a1", "a2", "ab1", etc. The problem is that the automated serialization engines are very fussy about names...

Oddly, this isn't unique to obfuscation; consider:

    public DateTime DateOfBirth {get;set;}

With automatically implemented properties, the field is supplied by the compiler (with an obscure name); but BinaryFormatter (etc) work against fields! If we make a supposedly innocent change to add validation logic (and so add our own field) our serialization can break.

Problem statement (with automatically implemented properties)

• We don't want out binary serialization to break just because we switch to manual implementation

Problem statement (with obfuscation):

• We want the type to be obfuscated
• We want a type to be serializable and deserializable in both original and obfuscated forms, ideally in a compatible way
• We don't want to write all the serialization code by hand (i.e. ISerializable)
• It would be nice (but not essential) if the serialized data didn't expose all our innards by name

For people using remoting or BinaryFormatter, this isn't simple. Formats like xml would work (assuming explicit xml names marked against each member), but xml isn't always the best choice - and you largely defeat the purpose of obfuscation with the attribute.

As it happens, all this is something that protobuf-net does for free! Since the contract doesn't involve names, we don't (usually) care what the fields/properties are called. It is also trivial to hook into ISerializable (if we need remoting), and WCF etc. Members are designated a numeric tag - for example:

   [ProtoMember(3)]
 public DateTime DateOfBirth {get;set;}

This then solves all our problems in one go; the only identifier used on the wire is the "3". There is a supported use-case that uses names to infer tags, but that is "opt in" (and not the preferred usage).

Plus of course; it is free, fast and gives much smaller binary output (for which Google must take much of the credit) - and works on virtuall all the .NET platforms. Give it a whirl ;-p