Reflections on Optimisation – Part 2, Existing Frameworks

Yesterday I introduced some topics on reflection performance, and investigated briefly the options to model abstraction, concluding that an abstract base-class / concrete subclass was a reasonable approach.

At this point, you could ask “so?” – the point is that I need a pretty optimised way of getting / setting data. So how about existing frameworks? Raw reflection? (PropertyInfo) ComponentModel? (PropertyDescriptor) HyperDescriptor? Let’s compare these options. I’ve setup a deliberately basic test rig, that simply tests each of these in a tight loop (again, deliberately avoiding boxing etc). As before, I’ll post the full code at the end of the series, but here’s the crude comparison:

Implementation	Get performance	Set performance
No wrapper (baseline) Base class HyperDescriptor PropertyDescriptor* PropertyInfo*

(numbers scaled, based on 750M iterations, except “*”; lower is better)

You might think “hey, PropertyInfo / PropertyDescriptor” aren’t too bad! Except for the *, which means that in order to get them into the same scale I had to introduce the line:

            count /= 100;

Where "count" is the number of iterations to perform. And even with this they are at the bottom of the table (by quite a margin). Yup – they really are over 100 times slower, which is why we need to optimise in the first place!

You might also conclude “HyperDescriptor doesn’t do too badly – just use that and move on”. Which would be valid in most cases (I’m pretty happy with that code, especially since that was my first foray into IL), except:

• It wouldn’t let me investigate field vs property, polymorphism, and the other common “is this faster” questions I mentioned in part 1
• It relies on constructs that don’t exist in CF / Silverlight, and which it would not be useful to duplicate (yet I need this for protobuf-net)
• Sometimes the extra crank of the optimisation handle matters

Summary

In this discussion, I hope I’ve demonstrated that there really is a problem just using PropertyInfo etc, that we can overcome (evidence: HyperDescriptor), but we would hope to do so in a simplified, minimal way that can be modelled (perhaps with slightly different implementations) on all the common frameworks.

Reflections on Optimisation – Part 1, Abstraction

Introduction

I want to take a few blog posts to discuss some ramblings on performance, reflection, and library code. The common “knowledge” is that reflection is slow and to be avoided, but this is missing the caveat “when used inappropriately” – indeed, as part of library / framework code it can be incredibly useful and very fast. Just as importantly, suitable use of reflection can also avoid the risk of introducing bugs by having to write repetitive code manually.

Between my musings on the Expression API, work on HyperDescriptor, and many questions on sites like stackoverflow, I’ve looks at this issue often. More, I need to do some work to address the “generics” issue in protobuf-net – so I thought I’d try to combine the this with some hopefully useful coverage of the issues.

The setup

What I am really after is the fastest way of getting values into and out-of objects, optimised as far as is practical. Essentially:

    var oldX = obj.X; // get
obj.X = newX; // set

but for use in library code where “X” (and the type of X) isn’t known at compile-time, such as materialization, data-import/export, serialization, etc. Thanks to my learning points from protobuf-net, I’m deliberately not getting into any “generics” answers from this. Been there; still trying to dig my way out.

I plan to look at a range of issues in this area, including (not necessarily in this order):

• types of abstraction
• fields vs properties vs polymorphism
• mechanisms for implementing the actual get/set code
• boxing, casting and Nullable-of-T
• lookup performance
• caching of dynamic implementations
• comparison to MemberInfo / PropertyDescriptor / CreateDelegate

Note that this is an area that can depend on a lot of factors; for my tests I’ll be focusing on Intel x86 performance in .NET 3.5 SP1. I’ll try to update when 4.0 is RTM, but I’d rather not get too distracted by the 4.0 CTPs for now. Even with this fairly tight scope, I expect this to take too many words… sorry for that.

First problem – abstraction

OK; so essentially we expect to end up with some API similar to reflection, allowing us to swap into existing code without re-writing it:

    var prop = SomeAPI["X"]; // perhaps passing in
                         // obj or a Type
object oldX = prop.GetValue(obj); // get
prop.SetValue(obj, newX); // set

This should look very similar to anyone who has used reflection or TypeDescriptor. And I plan to compare/contrast to the default implementations. So: is there any noticeable difference in how we implement “prop”? This might give us a better solution, and it also impacts how I write the tests for the other issues ;-p I can think of 3 options off the top of my head:

• An interface with concrete implementation per type/property
• An abstract base class with concrete implementations per type/property
• A sealed implementation with delegates (of fixed delegate types, one matching the get, one the set) passed into the constructor

There is no point introducing any dynamic code at this point, so I’ll test each these implemented manually (deliberately avoiding boxing), compared to a direct wrapper (no abstraction), and direct access (no wrapper, as per the original introduction):

I’ll post the full code later (and add link here), but this gives the initial results:

Implementation	Get performance	Set performance
No wrapper (baseline) No abstraction Base class Interface Delegates

(numbers scaled, based on 750M iterations; lower is better)

The conclusion of this is that, for the purposes of representing a basic property wrapper mechanism a base class is just as good as a wrapper with no abstraction – so we should assume a base-class implementation. An interface is a little slower – not enough to be hugely significant, but a property wrapper is a pretty specific thing, and I can’t see the benefit in making it more abstract than a base class.

For the purpose of adding context, the (very basic) code I’ve used to test this base-class approach is simply:

Summary

It might not seem like we’ve come far yet; but I’ve described the problem, and narrowed the discussion (for the purpose of micro-optimisation) to implementations using a base-class approach. Which is a little annoying on one level, as the delegate approach (via DynamicMethod) is in many ways more convenient.

Crimbo Tooling Presents

As part of development, I always try to keep my eye on any tools that make my life easier. Sometimes that means IDE tools – and sometimes more general tools. I just wanted to share the things that have helped me recently. Maybe you’ll decide to treat yourself for xmas ;-p

VMware

Recently, Player 3 (free) and Worstation 7 (£$€) were released; any developer not using virtual machines regularly is missing some serious tricks – and now that the free “Player” product allows you to create machines (bad news for EasyVMX?) there is no excuse not to. I compared it (informally) to the Windows (7) Virtual PC, and I’ve found Workstation much nicer to get along with. I honestly can’t quantify why (except for the really annoying pause whenever I resized a console in WVPC), but it just feels slicker.

Fences

Desktop icon management; everyone I’ve shown it to simply loves it (and they are still using it). Fences is, in some ways, a flashback to Program Manager (for those of us who remember Windows 3.x).

You know how your desktop accumulates cruff? Fences lets you group them – both visually and logically (drag groups instead of individually). And perhaps more importantly, hide/show them all (or your choice of them) in a single action (double-click on the desktop.

Some people might (and have – and no, I don’t have that many!) criticize me for keeping anything on the desktop – but it is the biggest UI element I have; why should I ignore it? It lets me keep things in logical groups, uncluttered, and easy to hide / show. Just lovely.

MaxiVista

When I’m in the office, I switch from my laptop to my workstation (more “grunt”, fixed dual head, etc) – but my laptop is still useful. Thanks to the joys of RSI, and my freaky keyboard/mouse setup, switching between the two is a pain. A KVM is an option, of course, but involves messing with wires, and doesn’t let me do everything at one. Step up MaxiVista! This great bit of software lets me configure my laptop as a monitor for my desktop PC – either as an extended desktop or for remote control (and I can toggle between the two with a single key, conveniently placed (on my freakboard) next to the Windows key.

Extended desktop: as you expect; drag a running app off the fixed monitors, and it appears on the laptop. It works brilliantly; sometimes (not always) there is a tiny bit of latency (similar to remote desktop etc, which makes a lot of sense), so I mainly use it as the monitor with static content (the requirements / spec, for example) rather than the monitor that I’m using for active UI work.

Remote control: in another mode, move the mouse off the monitor and it takes charge of the laptop; no noticeable latency now, as it is just sending the mouse movements and keyboard, and I find this fantastic, for example, for cross-OS testing (my laptop and desktop run different OS). No messing with KVM, VM, or RDP – just move to the computer you want.

There is also desktop mirroring for showing the same desktop somewhere else; I haven’t needed this yet, though.

It supports up to 4 remote monitors, and works alongside your existing multi-monitor setup, so you can go mad. I don’t think I could gainfully use more than 3, but hey.

Got Mono?

This week, a lot of new shiny Mono tools were released. Mono has always been something on the edge of my awareness, but I haven’t really had the time to get into it fully, in part because the tooling was… “incomplete”. I don’t mean that as an insult; merely that for most Windows devs, Visual Studio would be more attractive.

First: I’d better clarify my position. I really like Visual Studio; it works great for most of what I need – but with some cross-platform open source projects under my belt, and being aware of the common “community” (stackoverflow etc) questions about cross-platform, I now many of the pain points. I’m happy that both set of tools can co-exist without anyone getting upset.

The main time I started getting into Mono was for protobuf-net – I've had a lot of positive feedback from people pleased that it works on Mono, but my tooling for testing it on Mono has always been poor.

Step up MonoDevelop; with the new 2.2 release, IMO this actually starts to become a reasonable proposition for regular development (maybe not winforms…). With support for (among others) ASP, ASP.NET MVC, Moonlight and Gtk# – and inbuilt SVN and NUnit support it shouldn’t be dismissed. I’m in the process of ensuring that protobuf-net works (and passes the tests) in both IDEs. There are also spikes around that show Android development on MonoDevelop.

Additionally, Mono 2.6 introduces much better support on a wide range of APIs – most notably borrowing the Silverlight WCF stack, and things like the DLR. My only problem now is figuring out how much “old” Mono to support. I’m quite tempted to make 2.6 my new baseline, for simplicity.

Of course, that isn’t all – with MonoTouch for iPhone, and Mono Tools for Visual Studio, the Mono eco-system is truly starting to come into its own. I plan on spending a lot more time getting to know it more than I already do, and have proposed a session to present at ddd8. Interesting changes.

Red Gate and the Holy Grail

I’ve been playing with the EAP version of Red Gate’s .NET Reflector lately, which is a very nice addition to already excellent Reflector that we all love. I hadn't opened VS2008 on my test rig for a while, but did today... the “update thyself” message is pure Pythonesque joy:

Anyway, the EAP is discussed here, with the current build always near the top on the support forum. It certainly makes my life easier when debugging! (caveat: note the warnings in the posts – i.e. don’t go crying to Red Gate if it breaks your computer, pawns your car, or takes the cat to a tattoo parlour “for giggles”).

Solving Delegate Variance

Post redacted. It turns out I was making a meal out of something silly - converting from (for example) an EventHandler delegate-instance to a MouseEventHandler delegate instance.

I did it the hard way; Barry Kelly has the right idea (see comments). I bow to greater wisdom.

Does it work on MonoTouch? And more crazy constructors…

A double-feature ;-p The first didn’t seem to warrant a blog entry by itself, but I found if fascinating; as if interface constructors aren’t enough… when is “new T() == null” – and I’m not accepting Nullable<T> ! Ayende Rahien posts a fascinating “feature” of .NET – well worth a read. Caution – it might make you feel dirty…

But what I really came here for; MonoTouch… I had an e-mail earlier from somebody asking if protobuf-net would work on MonoTouch. My answer there is simple: I have no idea! I’d love to be able to give a better answer, but even the free evaluation edition needs a Mac. protobuf-net is a voluntary and revenue-free project, and I simply can’t justify buying a Mac to investigate.

I’m pretty sure that the “Reflection.Emit” code would have to go, but that is already the case for Compact Framework etc. I honestly don’t know about the rest of the reflection and/or optimisations. I might be able to offer guidance if anybody wants to try to get it working on MonoTouch (unless you want to lend me your Mac, iPhone and MonoTouch license for a while…).

LINQ Query Syntax – a whistlestop

Based on some feedback (both comments and e-mail) to my last entry, it seems obvious to me that a number of readers aren’t familiar with the way that LINQ maps query syntax to real code.

So I thought I’d offer a very brief taster. For real detail here, good resources include:

• the spec (§7.15.*) – or maybe the annotated spec
• C# in Depth (or 2nd Edition)

So hang on – I said this was a whistlestop, but this isn’t a short blog. You see, I never said this was a simple area. Go read one of the above and hopefully it will make more sense… but we’ll muddle on and see if anything sticks…

What is LINQ query syntax?

At the spec level, it isn’t quite what many people think it is… it knows nothing about what it is doing. It doesn’t have any knowledge of “IEnumerable”, “IQueryable”, etc – it is just a set of rules that map keywords like “from”, “select”, “join” etc into instance-style method calls using lambdas.

The rules for this are… “complicated”, for example:

A query expression with a second from clause followed by something other than a select clause:

from x₁ in e₁
from x₂ in e₂
…

is translated into

from * in ( e₁ ) . SelectMany( x₁ => e₂ , ( x₁ , x₂ ) => new { x₁ , x₂ } )

Don’t worry – you don’t usually need the details; I didn’t use the spec to write my custom “SelectMany”, for example. A bit of guesswork, and a quick check against the standard “Enumerable.SelectMany” got me going.

So what does my code look like?

As always, reflector will be useful (if you crank down the optimisations, as discussed here), but let's pick apart the example from yesterday:

The first “from” is largely inert; perhaps the most important thing it does is propose a name (and optionally type) for the first variable* “path”.

*=I’m using the term loosely.

The additional “from” statements are more interesting. These do a “SelectMany” (see quote above), but in order to keep “path” in scope, it uses the overload that accepts two lambdas; one to select the data (“File.OpenRead(path)” etc), and another to create a new type representing both “path” and the new data. Not necessarily very obvious, but something like:

(The “x1” / “x2” identifiers are introduced by the compiler)

So where do the methods come from?

Here’s the interesting thing: the LINQ spec doesn’t care! It could be an instance Select (etc) method declared on the actual type that happens to be “in play”; it could be a custom extension method provided after-the-fact. The actual method is resolved by regular C# member resolution rules. A few interesting examples of alternative implementations include:

• LinqBridge – LINQ-to-Objects with .NET 2.0/3.0
• Visual LINQ – watch queries as they happen
• PushLINQ in MiscUtil
• The .NET Reactive Framework
• PLINQ

At one point I had an F#-style async implementation using LINQ query syntax, but… well, I wasn’t very happy with it – sometimes you can push too hard ;-p

Or (as in the example with IDisposable) you can just offer some new code for specific cases. As long as the C# compiler can resolve a preferred implementation, you’re in business.

Summary

• LINQ = translation to instance-style method calls using lambdas
• You can write your own
• It isn’t at all limited to things like IEnumerable
• Since it uses lambdas, it can be delegate-based or Expression-based

Go crazy; try your own query syntax implementation today ;-p

SelectMany; combining IDisposable and LINQ

Something that often trips me up in LINQ is that you frequently have to break out of the query syntax simply to add a few “using” blocks. It is important to keep the “using” to ensure that resources are released ASAP, but the regular LINQ query syntax doesn’t let you do this. For example (and I realise there are better ways of doing this – it is for illustration only):

Trying to write this (or something like this) in LINQ query syntax (or fluent syntax) just gets ugly.

If only there were a way to convince it to run the “Dispose()” for us as part of the query! A “let” looks promising, but this won’t do anything special in the error case. But there is a pattern (other than just “using”) that lets us conveniently run code at the end of something… “foreach”. And both query-syntax and fluent-syntax already play nicely with extra “foreach” blocks – it falls (loosely) into “SelectMany”.

Thoughts; we could write an extension method that turns “IDisposable” into “IEnumerable”:

This is a step in the right direction – but those hanging “Using” blocks bother me - and the disposable object isn't quite inside the iterator - there are probably some corner-cases that would lead to items not getting disposed. But LINQ syntax is purely a query comprehension syntax; we aren’t limited to the current conventions – we can write an alternative version of “SelectMany”:

(hey, I never said SelectMany was pretty!)

This allows a much more interesting use:

This now does everything we want, without any ugly – our disposable items get disposed, and we can write complex chained queries.

I still haven’t decided whether this is too “out there” – i.e. whether it adds more benefit than it does confusion. All thoughts appreciated!

State Assassins with experience of Reporting Services

No, this isn’t one of the more unlikely job listings on careers – instead, a bit of work from last week that I found pretty interesting.

I was essentially adding some consultancy input to an unfamiliar codebase; the problem was that the site would be running very smoothly, then page-times would go crazy for a while, and then suddenly go back to normal.

It turned out (after putting state into SQL-Server for analysis) that some of the state records were unexpectedly large. Damagingly huge. So what were they? After all, the app only used state to store one tiny innocent object…

One advantage of using a database for state is that you can get the data back out, and it turns out that it isn’t too hard to reverse a BLOB back into the state objects (it helped that stackoverflow had half the answer already).

The cuplrit? You might guess (from the post title): reporting services and the web-forms report viewer control. As soon as a user viewed a report, it was using state to cache the data – which might work great for small reports, but if the report is large it gets… fun.

Perhaps a design failing in ASP.NET state; once a large object is in state, it goes up and down the wire (between the web-server and the state storage) on every state-enabled request for that user, even when no longer using that data. So if they used the report 10 minutes ago, and are now just looking at nice slim tidy pages, they are still dragging a huge state object over the network all the time, potentially crippling the site for lots of other users.

If the page had been highly AJAX-enabled it could have been much worse. And the reason the problem ended so abruptly? My reasoning is that the user in question (who is harming the site through no fault of their own) got bored of sluggish responses and did something else for a while.

Learning points:

• Even if you aren’t using SQL Server for state normally, it makes analysis much easier as a temporary swap-out
• Watch out for web-controls (especially reports) that might (ab)use state
• You can decipher a regular state BLOB – it isn’t tricky (I’ll update that SO question with an example if anyone is interested)
• Consider partitioning your app, or disabling state in some scenarios
• We aware of how much data your reports are dragging around with them

Controlling WCF / protobuf-net at the Endpoint

protobuf-net has had WCF hooks for a long time, but (to be frank) they’ve been a tad…. flakey.

Firstly, if you use “mex” generated proxy classes, it has a habit of discarding key metadata (the “Order” of the [DataMember]). This can be resolved either by using a shared DTO assembly, or by using fixups in the partial class – but…

Secondly, even if you have a shared DTO assembly, if you use the standard “add service reference” (or svcutil.exe), it duplicates the service-contract interface, and it drops the [ProtoBehavior] marker. Again, you can get around this by using the WCF channel directly or subclassing ClientBase<T> – but it would be nice if this wasn’t necessary.

Additionally, it would be very nice if the protobuf-net aspect was configurable (app.config/web.config); allowing greater reuse, and allowing parallel endpoints (with/without protobuf-net, to support vanilla and protobuf-net aware clients).

The community to the rescue

Step up Scott Prugh, who an embarrassingly long time ago very kindly supplied me with all the cogs to make this happen. Apologies for the delay, but things have been hectic and when I tried to test it my WCF brain-cells were switched off (in short: I borked my tests, couldn’t get it working, and had to shelve it for a while).

So what’s the shiny? Simply: rather than decorating your operations with [ProtoBehavior], we can apply it in the configuration instead, by adding an endpoint-behavior that references a custom WCF extension (which we must also add). This sounds worse than it is'; inside “system.ServiceModel” (in your app.config/web.config, this means adding a few lines of boilerplate:

And then to both client and server, we simply specify the endpoint configuration:

(in more complex scenarios you might want to add the custom extension to your existing endpoint configuration).

What’s the catch?

In the most part, that’s it. WCF will use protobuf-net for any suitable objects (data-contracts etc). Note that this is a coarser brush than the per-operation control, though (you could always split the interface into different endpoints, of course).

Also, protobuf-net does have some subtle differences (especially regarding empty objects), so run your unit tests etc.

Note that it only works on the full-fat WCF; it won’t help Silverlight etc, since it lacks the extension features – but that isn’t new here.

Finally, the resolver in WCF is a pain, and AFAIK wants the full assembly details including version number; so one more thing to maintain when you get new versions. If anyone knows how to get around this?

plzsendtehcodez

A minimal (but working) example is in the repo (client, server). This uses a shared DTO assembly (to avoid the need for fixups) and is based on the WCF project template. I’ll try to go back and retrofit the Northwind example shortly, and perhaps include the WCF samples in a simple download.

Highly Polished Software

No, not what you think...

Is it just me that dreads finding that the office cleaners have come around? Re-attaching jostled cables and re-adjusting monitor heights is one (annoying) thing - but how much polish does a desk really need? I can barely touch my laptop without it sliding over the desk - so you can imagine what my "freak keyboard" is like.


Looks like I'm going to have to spill copious amounts of coffee just to make the desk usable again. Sigh.

Express 2010; Basic and Expert Settings

Or: “Where has Start Without Debugging gone?”

I'm a fan of C# Express; it is great tool for quick/scratch code, and if you find my machine without at least one Express window in the task-bar, then it must be Patch Tuesday.

It looks like the Express editions of 2010 have different menu configurations for basic and advanced users:

This makes a lot of sense, but caught me off guard - especially since many of the key chords work in both modes.

I won’t attempt to list all the differences; but if you can’t find your menu items… try switching to Expert mode.

Lots of 2010 Shiny, and me feeling stupid

Lots of changes are underway:

• MSDN revamped
• VS2010 beta 2 available (and launch date announced)
• MSDN subscriptions simplified
• TFS now also for small teams

And because you'll want somewhere to play with beta 2:

• Windows Virtual PC RTM
• Security Essentials RTM

In particular, beta 2 now includes the “Express” line-up, and TFS now supports installation on client OS and has a simplified install; maybe getting a CI server (on just the Microsoft stack) will soon be practical for the individual / small team.

Feeling stupid – moving files between guest and host

I’ve downloaded the new VS2010 line-up – onto my host OS; obviously, this is beta software with the potential for mayhem (or more likely: not be 100% uninstallable), so I want to install it on the guest. In case you haven’t tried Windows Virtual PC yet (perhaps you haven’t got Windows 7 yet), then note that it loses the Microsoft Virtual PC feature to drag files between host and guest. And my guest can only really see http (no local network). So how to get the files there?

I sat there looking silly… USB was an option (Windows Virtual PC can grab USB devices), but seemed overkill. It turns out that the answer is blindingly obvious. So obvious that it didn’t even occur to me to try it. Copy… Paste. D’oh! (this does need the integration features enabled, but that isn’t a problem).

So there you go; I’ll admit to being a muppet, and hopefully it’ll save somebody a few minutes wondering.

Anonymous Type Tricks - a clarification

The other day, I posted about using anonymous types to pass multiple pieces of state into a method - reminder:

byte[] resp = client.Post(destUri, new {
id = 100021,
dob = DateTime.Today,
caption = "Hello world"
});

About the same time Ayende Rahien posted about a related but different use in Entity Framework - example:

.Case(
e => new {
  manager = e.Manager.Id
  thisIsADiscriminator = “E”
}
)

There is an important distinction between the two; in the first example, the property names (in the anonymous type) are only relevant to the caller. In the second (EF), the names are relevant to the callee. This is a huge difference. I don't know enough about EF to say more, but just a note: if you are doing something like this for your own types, please use a named property object to pass expected state into a method. Think of the kittens.

Novel uses of anonymous type initializers

On a related note; there was an interesting stackoverflow question today that shows another use of the anonymous type syntax - for specifying multiple members in a type-safe way:

IncludeProperties<IUser>(u => new {
   u.ID, u.LogOnName, u.HashedPassword });

The important thing to note is that the above is an Expression (not a delegate), and the code inside tears the Expression apart to find the properties that we are interested in. It never actually invokes the code as a delegate, nor does it ever actually instantiate the anonymous type. It might just be me, but I thought that was quite cute ;-p

Go! Dynamic

.NET 4.0 is looming (lurching?) ever-closer, and one of the many interesting developments in this release is the mainstream DLR. I’ve been trying very hard to find time to get into this, but I’ve failed horribly – every time I get some spare time, something comes up. Fortunate, then, that there are plenty of code-geeks around.

In particular, I urge you to have a look at Michael Foord’s recent articles; in particular Python for .NET Programmers (but a few more here). Rest assured that there is also plenty more in both “IronPython in Action” (from the IronPython side), and books like (not yet complete) “C# in Depth, Second Edition”  (from the C# consumer side of “dynamic” – lets face it, I’m primarily a C# developer).

I’m sure that Michael could suggest a hundred other uses, but I’ve got some configuration script scenarios in mind that I really need to find time to investigate.

Pass data simply; learning from jQuery and ASP.NET MVC

One of the really nice things that is common to both jQuery and ASP.NET MVC is how it passed multiple values around. They both avoid the complexity of things like dictionaries (or alternatively forcing specific types) by letting the caller use ad-hoc objects; by which I mean either a jQuery “post” method like:

    $.post("test.php", { name: "John", time: "2pm" } );

Or an ASP.NET MVC ActionLink of the kind:

    <%= Html.ActionLink(category.CategoryName,
 new { action="List", id = category.CategoryName})%>

In both cases, we are passing caller-specific data (not callee-specific data, so optional / named parameters wouldn’t help), but without the pain that we might have used in the past. Of course, we could also pass it one of our domain entities, and as long as the names all matched up, it wouldn’t care. Which is nice.

I’m going to concentrate on the second of these mainly – this is a hugely versatile tool; and while it shouldn’t be used for every job, it sure is handy. Here’s an example of using the same approach for a more readable Format syntax:

    string s = Format("You are {age} years old and your last name is {name}",
     new {age = 18, name = "Foo"});

There are a myriad of opportunities for this type of use. My main aim here is just to show some of the possibilities – and let your collective imaginations run wild. For example, to bring the jQuery “post” approach into regular C#:

    using (WebClient client = new WebClient())
 {
     byte[] resp = client.Post(destUri, new {
         id = 100021,
         dob = DateTime.Today,
         caption = "Hello world"
     });
     string html = client.Encoding.GetString(resp);
 }

With supporting code:

    public static class WebClientExtensions
 {
     public static byte[] Post(this WebClient client,
         string address, object values)
     {
         if (client == null)
             throw new ArgumentNullException("client");
         if (values == null)
             throw new ArgumentNullException("values");

         NameValueCollection valueLookup =
             new NameValueCollection();
         foreach (PropertyDescriptor prop in
             TypeDescriptor.GetProperties(values))
         {
             object val = prop.GetValue(values);
             string sVal = prop.Converter
                 .ConvertToInvariantString(val);
             valueLookup.Add(prop.Name, sVal);
         }
         return client.UploadValues(address,
             WebRequestMethods.Http.Post, valueLookup);
     }
 }

Sure, this is just a basic example and there are lots of things I could do to provide more flexibility, but it shows the key points.

Numeric keypads; what is the point?

As I hinted the other day, I've recently had some stress/RSI symptoms in my arm. Nothing too serious, just definitely worth fixing ASAP. I'm by no means crippled with RSI or anything, but I am on a mini-crusade for programmer-health, and want to help others avoid unnecessary stress / irritation.

I had a very productive session with a physiotherapist today, who made some really constructive and pragmatic points (for example, demonstrating quite simply and clearly that the curve on my corner-desk points the wrong way, and simply swapping to a mirrored desk could help hugely).

But one startling point she made while observing me; as a programmer, I move frequently between the keyboard (in my case a decent ergonomic split jobbie) and the mouse (now a shiny new "vertical" model) - code a few lines, use the mouse to click a few buttons, test a few things via the UI - rinse, repeat.

Sure, I use the keyboard shortcuts, but I would struggle to remember the hundreds of magic key-presses and "chords" that drive Visual Studio.

Very rarely do I use the numeric keypad. Heck, I also use a laptop, and the only time I find myself looking for Num Lock is when my password inexplicably doesn't work. Yet every time I move between keys and mouse I have to travel this pointless distance.

Lets not do that

Numeric keypads; absolutely useless. Underused, oversized, badly placed, and they don't agree with 'phones on which way the numbers should go.

She gave me a tip to simply try a keyboard without one, letting me have the mouse much closer. You can pick up a "Goldtouch" ergonomic keyboard (sans keypad) for about GBP25 at Amazon. This is so simple and so obvious that I'm just stunned that people insist on still including it, even on "ergonomic" keyboards. If it is so ergonomic, why is it causing unnecessary travel?!?!

If you want a keypad (and I do find them handy occasionally), then buy a USB keypad (like they sell for laptops) - roughly GBP10, and that is for the models that double as a handy USB hub. If you're right-handed, stick it on the left, or on the far side of the mouse - whichever you prefer and find most comfortable. Plus you can move it to either side if a "sinister" shows up at your desk.

Join the revolt! I invite you to ditch (or move) your keypad today!

protobuf-net vs NetDataContractSerializer

In a user-group meeting yesterday, I was talking briefly (in a “grok” session) about “protocol buffers” (protobuf-net in particular), and I was asked a question about the comparison to NetDataContractSerializer. I didn’t have the answer to hand (I only had numbers vs DataContractSerializer), so promised to find out…

First; note that the key difference between DataContractSerializer and NetDataContractSerializer is (as I understand it) that NetDataContractSerializer includes more type metadata. This makes it possible to do a few things involving “object” and unanticipated subclasses (which can be useful), but renders it .NET-specific (and less version-tolerant).

The test

As per my existing test rig, I test 1000 iterations of a moderately large extract from the Northwind sample database (into some pretty-standard POCO types); I simply extended this rig to incude NetDataContractSerializer. For completeness, I have 3 variants of the tests, tested separately each against a different set of serializers:

1. LINQ-to-SQL generated classes with serialization enabled at the data-context; suitable for protobuf-net, DataContractSerializer and DataContractJsonSerializer
2. as the first test, but with the LINQ-to-SQL bits hacked out and [Serializable] added (to make it suitable for the core binary .NET serializers); suitable for protobuf-net, NetDataContractSerializer, DataContractSerializer, DataContractJsonSerializer, BinaryFormatter and XmlSerializer
3. as second test, but with ISerializable and IXmlSerializable implemented to support BinaryFormatter and XmlSerializer via protobuf-net

The numbers and analysis

Since only the second test supports NetDataContractSerializer, that is the one I’ll focus on (with results from the 3rd test included for interest only, marked *); here’s the numbers:

Serializer	Size (bytes)	Serialize (ms)	Deserialize (ms)
protobuf-net	133,010	10,769	23,511
NetDataContract Serializer	992,203	29,343	91,453
DataContract Serializer	772,406	16,272	66,755
DataContractJson Serializer	490,425	29,604	135,125
BinaryFormatter	276,366	67,151	56,253
XmlSerializer	1,043,137	29,257	38,528
BinaryFormatter*	133,167	12,095	24,653
XmlSerializer*	177,410	12,189	31,827

Some interesting points; firstly, for my sample data it is a myth that WCF’s binary type-centric serializer (NetDataContractSerializer) is going to improve things over xml (DataContractSerializer). Both in terms of bandwidth and processing time it is worse on every front. The biggest advantage of NetDataContractSerializer (in this case) is that the type model is more flexible in terms of interfaces, “object” and unknown subclasses; but this also makes it entirely non-interoperable. Of course, some details of the SOAP envelope used by the binary type-centric WCF transport may be more efficient – but I’m looking at payload for these tests.

Secondly (and fortunately for me) it yet again proves that google know how to do data transport; with protobuf-net being the clear winner for both bandwidth and processing time.

But for me, another point that I’d like to highlight is versatility and compatibility; the same classes can be used happily by a wide range of serializers (above), and presumably a few others too. For the record, the entities in the above test don’t even have any protobuf-net-specific attributes (the outer-most wrapper does have them, but only because at some point in the distant past I wanted a way to compare/contrast some protocol-buffers-specific details, which can only be controlled via the custom attributes).

Other thoughts

As with all performance tests, your specific environment and data may be an important factor. But based on the above numbers it may be reasonable to assume that protobuf-net stands a good chance of holding its corner.

Finally; following some unrelated conversations at the same user-group (they kept me pretty busy), I stress that tight serialization is only part of the story:

• It can help reduce bandwidth costs and CPU costs associated with (de)serialization, but it won’t help if latency is your issue; protobuf-net makes no claim to change the speed of light.
• While protocol buffers may be supported on a range of platforms, it is by no means ubiquitous; if your biggest demand is portability, perhaps use SOAP/WSDL (or offer both SOAP and protobuf-net on separate endpoints).
• By itself it won’t solve all your disconnected woes; although it can presumably be used in tandem with any message-queue based solution you might dream of.

The Most Important Programming Book

A holy war, you might think, for any of author / architecture / language. But for full-time professional geeks, how about a book that is much more important than any of these mere implementation details:

You might be thinking “huh?”, but as IT professionals (and programmers in particular), we are hugely exposed to RSI in our professional life. Don’t underestimate the risks; we do this job to pay the mortgage/rent, after all.

I switched to an ergonomic keyboard years ago (one of the Microsoft models with the split key groups), after a warning shot – but the importance of things like correct posture / equipment is huge. But my point here: don’t wait to be a sufferer: take preventative steps now to avoid problems (and pain) later. Even those fancy vertical / air mice are a snip compared to just a few days off work.

Stickers – now what to do with them?

For any stackoverflow users, the stickers are real – Jeff was kind enough to send mine directly, rather than wait for DevDays. I knew that ♦ would be useful!

Compulsory wooden-table shot:

They’re a little smaller than I expected, being only slightly longer than my mobile ‘phone. Now… if only I can think of what to do with them… so far, I’m wondering if they might make serviceable plasters (band-aid if that doesn’t translate). Or (edit) maybe a laptop-chassis emergency repair kit? (my last laptop survived 2 large cracks for a considerable time). Or I might drag them along to a user-group and see if there are any takers.

All other suggestions welcome…

Using reflector to understand anonymous methods and captured variables

C# 2.0’s ability to magically hoist “captured variables” into classes is often misunderstood, and there are a lot of tricky edge cases (especially when combined with the notorious “foreach” glitch).

So; how can we understand what is going on? Usually, the first tool to look at is reflector; so lets try with a simple example:

int div;
do { Console.WriteLine("Find numbers divisible by..."); }
while (!int.TryParse(Console.ReadLine(), out div));

int[] data = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
   11, 12, 13, 14, 15, 16, 17 };

var filtered = data.Where(num => num % div == 0);
foreach (var i in filtered) Console.WriteLine(i);

So what is really happening here? Load it into Red Gate’s .NET Reflector, and by default it doesn’t actually seem to help us much, ironically because it does such a good job of interpreting the code:

I’ll forgive it for interpreting “Where” as LINQ query syntax, since the two are 100% interchangeable. What we don’t want to have to do is look at the IL – it is, lets be honest, not fun – here’s a small fragment of it:

But if you look at the options in reflector:

you can choose the level of magic it applies. Lets try scaling it back to .NET 2.0 (a trick I use often to investigate the lambda Expression API):

Here, it has got rid of the query syntax (and would remove lambdas, too), and now uses the anonymous method approach (“delegate” etc), but it hasn’t really changed much (since C# 2.0 still allows captured variables in anonymous methods).

As an aside, it is very curious that it has left “data.Where” (rather than “Enumerable.Where(data)”) – I have left feedback with Red Gate ;-p

So lets try .NET 1.0:

Now we’re talking! Here we can clearly see the class (<>c__DisplayClass1) and local variable / instance (CS$<>8__locals2) that is used to do the hoisting, the hoisted field (CS$<>8__locals2.div) and the anonymous method (CS$<>8__locals2.<Main>b__0) that provides the logic implementation. All hyperlinked up for easy navigation - to the compiler-generated class; this is invaluable for anything non-trivial:

[CompilerGenerated]
private sealed class <>c__DisplayClass1
{
   // Fields
   public int div;

   // Methods
   public bool 
b__0(int num)
   {
       return ((num % this.div) == 0);
   }
}

Interestingly, we can also scale it back a level further to “None”; which shows compatible C# without candy like “foreach”. This is verbose for C#, but terse compared to the IL version; here’s the “foreach”:

The declaration isn’t in the above grab (it is at the top of the method), but this also highlights how the iteration variable (“i”) is declared outside of the loop, hence the “foreach”/”capture” issue.

Again, I find it quite fun that in this level of optimisation it doesn’t show “foreach”, but knows about extension methods; but I’m not complaining – it seems ungrateful to criticise such a minor glitch in such a wonderful (and free) tool. But for people using non-standard LINQ implementations (or just to be true to the optimisation level), it would be nice if it showed this appropriately.

So there you go; yet another way to use reflector.

Who says you can’t instantiate an interface?

(update update: the question has been raised: "should we use this in our code?" - and in answer: heck no! this is just a fun diversion around the edges of the C# spec; leave well alone...)

(update: I just realised that Jon Skeet mentioned this in passing here, too; I'll keep this here, as I think it is intriguing enough to warrant a more targeted post)

If I said that the following is valid C# (all the way back to v1), you might think I’ve been overdoing things:

interface IFoo
{
string Message {get;}
}
...
IFoo obj = new IFoo("abc");
Console.WriteLine(obj.Message);

But the funny thing is… this is actually legal! Or at least, it compiles and runs ;-p As you might expect, I’ve missed out some magic in the above ;-p

It turns out that there is a subtlety relating to how COM imports work… and it is possible to define against the interface a default concrete type. The C# compiler then interprets “new {InterfaceType}” as “new {DesignatedConcreteType}”.

Here’s the missing magic (including a concrete type):

class Foo : IFoo
{
readonly string name;
public Foo(string name)
{
    this.name = name;
}
string IFoo.Message
{
    get
    {
        return "Hello from " + name;
    }
}
}
// these attributes make it work
// (the guid is purely random)
[ComImport, CoClass(typeof(Foo))]
[Guid("d60908eb-fd5a-4d3c-9392-8646fcd1edce")]
interface IFoo
{
string Message {get;}
}

(see footnote) I'm struggling to find a mention of it in the C# specification (3.0), though... indeed §7.5.10.1 says:

"The type of an object-creation-expression must be a class-type, a value-type or a type-parameter. The type cannot be an abstract class-type.
The optional argument-list (§7.4.1) is permitted only if the type is a class-type or a struct-type."

So a non-legal feature? I'll let you be the judge...

Footnote: Mehrdad has clarified that the fact that it is omitted from the specification is OK because §17.5 allows any of of attributes in the System.Runtime.InteropServices namespace to break all the rules ;-p

Expression as a Compiler

It is no secret that I like playing with the Expression API. Well, I've finally spent some time collating some of my thoughts into a semi-coherent discussion on InfoQ. I hope it is useful to somebody.

Expression as a Compiler

Thieving rail

Not programming related…

Holy spaghetti monster! This morning I made the mistake of asking how much it will be to renew my annual rail ticket. I really wish I hadn’t asked… from GBP3117 to GBP3672 – which is very nearly 18% – yikes!

This isn’t a very original complaint, but have these people not seen the state of the economy! And when you consider that I have to pay this out what the state deigns to leave me after income tax (@40% on any new income) and state deductions (NI etc) – and by my crude reckoning, in order to pay this extra GPB555 I’d have to achieve extra income in excess of GBP1000 – just to break even on travel (i.e. before any other increases are paid for, you know, like inflation on everything else…).

If it wasn’t for my employer offering an interest free loan on travel costs, I’d have to add interest to the above bill (either interest lost on savings, or interest accrued on a credit facility).

So, my options:

• accept that any annual pay review that isn’t pretty extraordinary (unlikely in the current climate) is going to represent a pay cut
• drive instead and somehow convince my employer to let me work 2 (or more) days per week at home (otherwise it becomes even more expensive) – and sacrifice the time I spend reading / working /etc on the train
• look for a job with lower commute costs

I’ve been regularly commuting by train for nearly 4 years; renewing long-established travel arrangements should not be this stressful.

</rant>

GAC, Assembly Resolution , and all that Jazz

Microsoft seem to be big fans of the GAC - which is fine for them as the platform provider, but personally I prefer to keep my code away from the GAC as far as possible. A few pet reasons, including some still-healing wounds from COM/COM+ (dll hell), and the fact that most of what I write needs to get deployed to a server-farm by tools like "robocopy", which is not the friend of GAC.

This difference raised its head recently when I was trying to get the VS2008 tooling working for protobuf-net. The way it works is that as part of the VS registration (which is done in a VS-specific registry hive) you tell it how to get your package - and a valid choice here is to specify the codebase. This is the option I took, since it introduces no extra configuration details, other than the mandatory VS hooks.

So that is fine and dandy, dandy and fine. But problems soon happened. In particular; since I already had dlls for protobuf-net and protogen, I wanted to ship those in the install directory, and have my VS package simply reference them. Which turned out to be not-so-simple...

Assembly resolution

The first thing to break down was assembly resolution; my package would simply refuse to find the dlls. Because of the different paths etc, your package's isntall directory is pretty unimportant to VS. If your required dlls are in the GAC, then it will find it - but I really didn't want to do that. But there are other tricks...

A little-known feature in .NET is that there is an event (AppDomain.AssemblyResolve) that fires when it can't find an assembly. By handling this event we can load it manually (for example: from the file system, download-on-demand, or from an embedded resource):

This works fine in most applications, but not VS; this event simply never fires within a VS package. Oh well. Another feasible option is to use reflection to load the dll (probably acceptable if the API is very shallow):

 

Not pretty, but it'll work - except now you just move the problem downstream; any dlls that our library needs will be missing too, with all the same problems plus some more issues associated with the "load context" (depending also on the whole LoadFile / LoadFrom choice)... oh dear.

Processes vs AppDomains

Our original problem was that files that existed locally weren't being resolved due to the assembly resolution paths of VS; but what if the second assembly was actually an exe? In the case of protogen.exe this was fine. So we could launch a new Process (setting the working path, etc); not elegant, but very effective - in particular it bypasses all the issues of the "load context" inherited from VS; we have our own process, thanks very much.

But there is an alternative... a second AppDomain; this lives within the same process, but is isolated (in terms of assemblies, configuration, etc), and can be unloaded. And neatly, you can execute an assemblies entry-point very easily (conveniently passing arguments as a string[] rather than having to pack them down as a single string for Process):

This is actually quite a cute way of launching a sub-process (for .NET code).

Processes within AppDomains within Packages

As it happens, in my case my child dll itself has to launch a Process during execution ("protoc.exe"); normally this is fine, but VS was soon causing problems again; for some curious reason (presumably linked to security evidence), when using ExecuteAssembly this inner Process.Start would always flash a console frame (even when told not to). Oddly, if I instead launch a Process from my Package, and that Process launches a Process, then this doesn't happen. A bit of an undesirable quirk, but as a consequence this multi-level Process tree is currently how the VS package does its work.

But in general you can use ExecuteAssembly as a neat alternative to a Process.

Summary

VS packages have "fun" written all over them... maybe my borked installer was right all along ;-p

Busy busy busy

Just a quickie...

Apologies if you've sent me a patch, bug, or other piece of work over the last week or so; I will get to it, but things are a bit hectic at the moment; "crunch" time on a project for work, plus lots of non-work things keeping me very busy.

So I'm not ignoring you; I'm just completely maxed. Hopefully normal service will resume shortly ;-p

Things not to do with an installer...

First - important point: this bork never left my laptop; this is not a problem in the package I mentioned yesterday!

Can you spot the face-palm in the following (that installs the protobuf-net Custom Tool into Visual Studio 2008):

That will install just fine, and the Custom Tool will work. And then you uninstall it.... oh dear. It helps if you know that {FAE04EC1-301F-11D3-BF4B-00C04F79EFBC} is Visual Studio's nickname for "C#". Did you notice that innocent little "DeleteAtUninstall" flag? Somehow it get set to true (it certainly wasn't deliberate).

Oh well, perhaps I can get by without all the inbuilt tools! Or maybe I need to rebuild my crippled registry... Fortunately this only impacted my laptop before I spotted it (I do test things, honestly!).

Still - it could have been worse; a few levels higher up and I could have completely broken Visual Studio (or potentially windows, but I would hope that the operating system wouldn't let me do something truly stupid, such as deleting HKLM/Software).

The moral of this: when you are writing installers, double check both the install and the uninstall do exactly what you expect. No less, and (perhaps more importantly) no more.

protobuf-net; now with added Orcas

A long running gripe with protobuf-net is that while the code generation (from .proto) works, it isn't easy to integrate into your development process. Well, wait no longer! I'm very happy to announce that protobuf-net now has Visual Studio 2008 support

(important note: this does not include the "Express" editions of Visual Studio).

A big vote of thanks goes to Shaun Cooley, who kindly created me the initial "cut" which I've extended. I did try this before, but I drowned in the complexity that is Visual Studio extensibility.

So what do I get?

Initially, the installer includes a "Custom Tool" (registered against the .proto extension), and an item template (not currently registered by the tooling):

The Custom Tool

This means you can add new text file called "mymessage.proto", and it will automatically start running C# (see below) code-generation on that .proto - writing a .cs file to match your .proto. This includes things like error-processing, so if you have errors in your .proto you can click on them and it will take you to the right place in the file. It will also add the protobuf-net reference if you don't already have it.

For existing files, you can set the Custom Tool manually; the string you need is "ProtoBufTool" (original, huh?).

At the moment, protobuf-net lets you pass a number of options on the command line. This same functionality is available in the IDE too - just (ab)use the "Custom Tool Namespace" - it accepts a semi-colon delimited set of arguments (the first being reserved for the actual namespace to use if your file doesn't specify a package) - i.e. if you use the string ";fixCase", it will apply C#-style case fixups (so "name" becomes "Name", etc). The arguments are exactly the same as on the command line tool (protogen.exe).

And if you think this is a bit ugly; I have "plans" (see below).

The template

The installer also includes (in the loosest sense) a custom item template, so you can use Add -> New Item to get a new .proto file with a suitable skeleton (including things like the project's namespace, etc). It doesn't currently install this template - and I'll be honest; this is simply because I haven't yet had time to crack the installer specifics of this (it is.... fiddly).

To install this manually:

• copy the BrotoBuf.zip file from (protobuf-net install dir)\ItemTemplates\csharp\data\1033 into (VS install location - Microsoft Visual Studio 9.0)\Common7\IDE\ItemTemplates\CSharp\Data\1033
• from the VS2008 command prompt, execute "devenv /installvstemplates" (it may take a minute or two)

Enough; where is it?

The installer is available from the project home page.

What next

The support at the moment is limited; I have a number of things I want to do:

• Add VB support! (the main problem here is detecting the project's language; the code generation itself works)
• Register the ItemTemplate during installation
• Move a lot of the command-line switches into custom .proto attributes
• Fix the annoying UTF8 BOM issue once and for all...
• etc

I hope you find it useful; working with VS extensibility has certainly been a different set of challenges - I'll blog about some of those separately.

Using Expression when creating runtime types

In .NET 3.5, the framework provides the (greatly appreciated) mechanism to compile an expression tree to a delegate - providing a convenient way to write dynamic methods without having to dabble in IL (never pretty).

Over the weekend, I've been looking in more depth at the DLR trees in 4.0 (beta) - and an interesting little nugget leapt out at me... you can now compile an expression tree into the body of a MethodBuilder! This means that even when writing a full type at runtime you can write the methods using an object model rather than Reflection.Emit.

My current example of this doesn't really illuminate much, so I won't post it "as is" - probably to follow. But does this sound "time" for people having to learn how to use ILGenerator?

Don't get me wrong - expression trees aren't trivial, and you need to think carefully when writing them - but they are a lot simpler than writing IL directly.

Restricting attribute usage

.NET attributes are a great way of expressing metadata (when used appropriately), but I regularly see people frustrated that you can't have much control over when you can apply them. Sure, you can use [AttributeUsage] to limit it to any combination of fields, classes, properties, etc - but beyond that... nothing (unless you write an FxCop rule (hard work) or look at PostSharp (still hard work, and changes your build)).

I was looking at this today when somebody wanted to restrict the usage of an attribute just to members in a particular type (inheritance) hierarchy, and I trotted off the usual "sorry, you can't do that" response... but then I went and spoilt it all by proving myself wrong; actually, you can do this - using one of the rarer features in C# - protected classes.

In C#, you are only allowed to use the protected modifier (on a type) on a nested type - and the effect is that only types that inherit from the outer type (and the outer type itself) can see the nested type. And interestingly, it works for attributes! Here's the example I came up with:

abstract class MyBase {
    [AttributeUsage(AttributeTargets.Property)]
    protected sealed class SpecialAttribute : Attribute {}
}
class ShouldBeValid : MyBase {
    [Special] // works fine
    public int Foo { get; set; }
}
class ShouldBeInvalid { // not a subclass of MyBase
    [Special] // type or namespace not found
    [MyBase.Special] // inaccessible due to protection level
    public int Bar{ get; set; }
}

An interesting trick...

Checking your career path

or... have you asked yourself lately: "what matters to me?"

In July 1999, the Microsoft platform consisted of a combination of Windows 9x, NT4, SQL Server 7 (or 6.5 in most cases), and VB6 (more correctly, Visual Studio 6). Monitors were deep, storage wasn't, and there was still often a funny slot in the front of beige boxes to take the ever-rarer floppy disk. More importantly, though, I stopped scrounging as a student and started getting paid to write line-of-business code using that suite.

The thing is, though; I'm still at the same company, with just a snitch short of 10 years under my belt. Sure, I've changed job title a few times since then - but I've also repeatedly heard it said that it is atypical (unnatural, even) to stay in one place so long these days (a "job for life" being ancient history) - especially as a first job.

But why should I change?

• I like the work and the team, and the projects are ethical; and usually varied, interesting and rewarding
• The company* is big enough to span most of the common "enterprise" scenarios for IT systems
• They have always respected that I'm happy being a code geek; I'm quite good at it, and I don't particularly want to be a manager (indeed, over the last few years they've introduced an additional grade of senior geek**, to respect this as a career progression option)
• We're pretty good about adoption of new technologies and methodologies
• I'm still learning and growing, both in work and on my own time
• My employer is liberal about my out of work community involvement - indeed, they actively encourage and support such
• They are pretty flexible, with a number of policies that help with the fact that (by my own choice) I live a stupid distance away and commute by train
• The compensation package is acceptable
• The sector (education) is relatively stable and secure

(*=if you are UK-based and want to check the recruitment boards, it is no secret that I work at RM)
(**=no, that isn't the actual job title)

My only possible concern is that as a result, I've been mainly limited to the Microsoft technology stack. I can't really complain here, though since:

• The stack is rich enough and fast moving enough that you can never know all of it
• As part of the senior technical team, I'm involved in our choice of tools, so it is (in part) of my own doing

So why should I change?

Frankly, I have no immediate plans to do so. I do, however, firmly believe that it is critical to regularly check that one is following the right path. I think that with the above list I've done enough to satisfy myself that I'm still in the right place for me (and conversely, for my employer: a malcontent employee isn't a constructive employee). But have you asked yourself the same question lately?

Some history: I was guilty of this a few years ago; we (my wife and I) found we'd been idly "drifting" through life for several (wasted?) years. We only realised this (the hard way) when my father was tragically killed in an accident while on holiday; in the sad, reflective months after that, the question(s) came up "what are we doing? where are we going? what is important to us?" As it happened, it turned out that going to work each day to pay the mortgage wasn't actually our life goal - and we made some big changes (including relocating half way across the country) and went the family route. We've never been happier. Of course, you need to make your own priority list (and it doesn't strictly have to relate to career), but make decisions - don't just drift idly. And "no change" counts as a 100% valid decision, as long as it is considered.

Summary

Reminiscent of a project health-check...

Priorities weighed and measured? check
Remedial action? none required
Follow up actions? re-book in 1 year, same attendees

And back to the grindstone...

protobuf-net support for 2.1

I've finally found time to add support in protobuf-net for the changes in the main protocol buffers 2.1 release.

For those interested in a fast, efficient, portable (interoperable), obfuscation-safe, binary serialization engine for .NET, then this adds (among other things):

• "packed" encoding for lists/arrays/etc of simple types (int, float, etc) - much more efficient, as it avoids the need to send a field identifier per element
  
• the ability (in .proto definitions) to mark members as deprecated (maps to [Obsolete])

It also introduces a range of bug-fixes, enhancements etc in my own code (nothing to do with 2.1) - including better namespace control when performing code generation.

Probably the biggest difference between protobuf-net and the other implementations is that although you can do contract-first (via a .proto), it doesn't force you to do that - so you can either define you types in a .proto, or you can just use your regular .NET classes with a few attributes (it can even piggy-back of WCF [DataContract]/[DataMember] in many cases). This has always felt more natural to typical .NET development (at least, from what I see).

I'm guessing it'll be a few months before the core Google API changes again, so hopefully this will be the last release (except for very minor bug-fixes) in that branch; I'm still trying to refactor (read: re-implement...) the code to work around those pesky CF limitations (and improve the performance and quality in the process).

Fun fun fun...

Exploring 4.0; Expression and Dynamic

You can't have missed that VS2010 beta 1 is now available. With the 4.0 release, most of the changes are in the framework dlls - there aren't a huge amount of C# language changes to pick up. I've done a little dabbling...

Expression

I looked previously at Expression on the CTP; well, guess what? They went and renamed everything... here's the previous example, updated; note the use of Expression.Block and Expression.Assign (rather than Comma and AssignProperty):

class Foo
{
   public string Bar { get; set; }
   public override string ToString()
   {
       return Bar;
   }
}
static void Main()
{
   var param = Expression.Parameter(typeof(Foo), "foo");
   var assign = Expression.Assign(
       Expression.Property(param, "Bar"),
       Expression.Constant("def", typeof(string)));
   var writeLine = Expression.Call(
       typeof(Console).GetMethod("WriteLine", new[] { typeof(object) }),
       param);

   var action = Expression.Lambda<Action<Foo>>(Expression.Block(assign, writeLine), param);
   var actionDel = action.Compile();
   Foo foo = new Foo();
   actionDel(foo);
}

It is also pretty clear that there are no plans to add language support for this (via a lambda); maybe something for the future.

Dynamic

I think "dynamic" is great if you are talking to COM or DLR; but should generally be avoided otherwise. Still; one interesting aspect is that it provides access to generic operators; which wasn't yet implemented on the last CTP. Well; it is here now - and it actually doesn't perform too badly... to avoid taking a dependency on an extra dll, I'd consider it...

int
Operator: 833ms (100000010)
Dynamic: 1383ms (100000010)

float
Operator: 786ms (3.355443E+07)
Dynamic: 1309ms (3.355443E+07)

decimal
Operator: 3974ms (100000010)
Dynamic: 3911ms (100000010)

As it happens the Nullable<T>performance isn't quite as good yet, but you can always use MiscUtil's Operator class ;-p

Other than that; unless I start writing lots of IronPython, I currently don't expect to use "dynamic" all that much.