Saturday, 15 August 2009

Using reflector to understand anonymous methods and captured variables

C# 2.0’s ability to magically hoist “captured variables” into classes is often misunderstood, and there are a lot of tricky edge cases (especially when combined with the notorious “foreach” glitch).

So; how can we understand what is going on? Usually, the first tool to look at is reflector; so lets try with a simple example:

int div;
do { Console.WriteLine("Find numbers divisible by..."); }
while (!int.TryParse(Console.ReadLine(), out div));

int[] data = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17 };

var filtered = data.Where(num => num % div == 0);
foreach (var i in filtered) Console.WriteLine(i);

So what is really happening here? Load it into Red Gate’s .NET Reflector, and by default it doesn’t actually seem to help us much, ironically because it does such a good job of interpreting the code:

image

I’ll forgive it for interpreting “Where” as LINQ query syntax, since the two are 100% interchangeable. What we don’t want to have to do is look at the IL – it is, lets be honest, not fun – here’s a small fragment of it:

image

But if you look at the options in reflector:

image image

you can choose the level of magic it applies. Lets try scaling it back to .NET 2.0 (a trick I use often to investigate the lambda Expression API):

image

Here, it has got rid of the query syntax (and would remove lambdas, too), and now uses the anonymous method approach (“delegate” etc), but it hasn’t really changed much (since C# 2.0 still allows captured variables in anonymous methods).

As an aside, it is very curious that it has left “data.Where” (rather than “Enumerable.Where(data)”) – I have left feedback with Red Gate ;-p

So lets try .NET 1.0:

image

Now we’re talking! Here we can clearly see the class (<>c__DisplayClass1) and local variable / instance (CS$<>8__locals2) that is used to do the hoisting, the hoisted field (CS$<>8__locals2.div) and the anonymous method (CS$<>8__locals2.<Main>b__0) that provides the logic implementation. All hyperlinked up for easy navigation - to the compiler-generated class; this is invaluable for anything non-trivial:

[CompilerGenerated]
private sealed class <>c__DisplayClass1
{
// Fields
public int div;

// Methods
public bool
b__0(int num)
{
return ((num % this.div) == 0);
}
}

Interestingly, we can also scale it back a level further to “None”; which shows compatible C# without candy like “foreach”. This is verbose for C#, but terse compared to the IL version; here’s the “foreach”:

image

The declaration isn’t in the above grab (it is at the top of the method), but this also highlights how the iteration variable (“i”) is declared outside of the loop, hence the “foreach”/”capture” issue.

Again, I find it quite fun that in this level of optimisation it doesn’t show “foreach”, but knows about extension methods; but I’m not complaining – it seems ungrateful to criticise such a minor glitch in such a wonderful (and free) tool. But for people using non-standard LINQ implementations (or just to be true to the optimisation level), it would be nice if it showed this appropriately.

So there you go; yet another way to use reflector.

Who says you can’t instantiate an interface?

(update update: the question has been raised: "should we use this in our code?" - and in answer: heck no! this is just a fun diversion around the edges of the C# spec; leave well alone...)

(update: I just realised that Jon Skeet mentioned this in passing here, too; I'll keep this here, as I think it is intriguing enough to warrant a more targeted post)

If I said that the following is valid C# (all the way back to v1), you might think I’ve been overdoing things:

interface IFoo
{
string Message {get;}
}
...
IFoo obj = new IFoo("abc");
Console.WriteLine(obj.Message);

But the funny thing is… this is actually legal! Or at least, it compiles and runs ;-p As you might expect, I’ve missed out some magic in the above ;-p

It turns out that there is a subtlety relating to how COM imports work… and it is possible to define against the interface a default concrete type. The C# compiler then interprets “new {InterfaceType}” as “new {DesignatedConcreteType}”.

Here’s the missing magic (including a concrete type):

class Foo : IFoo
{
readonly string name;
public Foo(string name)
{
this.name = name;
}
string IFoo.Message
{
get
{
return "Hello from " + name;
}
}
}
// these attributes make it work
// (the guid is purely random)
[ComImport, CoClass(typeof(Foo))]
[Guid("d60908eb-fd5a-4d3c-9392-8646fcd1edce")]
interface IFoo
{
string Message {get;}
}

(see footnote) I'm struggling to find a mention of it in the C# specification (3.0), though... indeed §7.5.10.1 says:

"The type of an object-creation-expression must be a class-type, a value-type or a type-parameter. The type cannot be an abstract class-type.
The optional argument-list (§7.4.1) is permitted only if the type is a class-type or a struct-type."

So a non-legal feature? I'll let you be the judge...

Footnote: Mehrdad has clarified that the fact that it is omitted from the specification is OK because §17.5 allows any of of attributes in the System.Runtime.InteropServices namespace to break all the rules ;-p

Tuesday, 11 August 2009

Expression as a Compiler

It is no secret that I like playing with the Expression API. Well, I've finally spent some time collating some of my thoughts into a semi-coherent discussion on InfoQ. I hope it is useful to somebody.

Wednesday, 5 August 2009

Thieving rail

Not programming related…

Holy spaghetti monster! This morning I made the mistake of asking how much it will be to renew my annual rail ticket. I really wish I hadn’t asked… from GBP3117 to GBP3672 – which is very nearly 18% – yikes!

This isn’t a very original complaint, but have these people not seen the state of the economy! And when you consider that I have to pay this out what the state deigns to leave me after income tax (@40% on any new income) and state deductions (NI etc) – and by my crude reckoning, in order to pay this extra GPB555 I’d have to achieve extra income in excess of GBP1000 – just to break even on travel (i.e. before any other increases are paid for, you know, like inflation on everything else…).

If it wasn’t for my employer offering an interest free loan on travel costs, I’d have to add interest to the above bill (either interest lost on savings, or interest accrued on a credit facility).

So, my options:

  • accept that any annual pay review that isn’t pretty extraordinary (unlikely in the current climate) is going to represent a pay cut
  • drive instead and somehow convince my employer to let me work 2 (or more) days per week at home (otherwise it becomes even more expensive) – and sacrifice the time I spend reading / working /etc on the train
  • look for a job with lower commute costs

I’ve been regularly commuting by train for nearly 4 years; renewing long-established travel arrangements should not be this stressful.

</rant>

Tuesday, 28 July 2009

GAC, Assembly Resolution , and all that Jazz

Microsoft seem to be big fans of the GAC - which is fine for them as the platform provider, but personally I prefer to keep my code away from the GAC as far as possible. A few pet reasons, including some still-healing wounds from COM/COM+ (dll hell), and the fact that most of what I write needs to get deployed to a server-farm by tools like "robocopy", which is not the friend of GAC.

This difference raised its head recently when I was trying to get the VS2008 tooling working for protobuf-net. The way it works is that as part of the VS registration (which is done in a VS-specific registry hive) you tell it how to get your package - and a valid choice here is to specify the codebase. This is the option I took, since it introduces no extra configuration details, other than the mandatory VS hooks.

So that is fine and dandy, dandy and fine. But problems soon happened. In particular; since I already had dlls for protobuf-net and protogen, I wanted to ship those in the install directory, and have my VS package simply reference them. Which turned out to be not-so-simple...

Assembly resolution

The first thing to break down was assembly resolution; my package would simply refuse to find the dlls. Because of the different paths etc, your package's isntall directory is pretty unimportant to VS. If your required dlls are in the GAC, then it will find it - but I really didn't want to do that. But there are other tricks...

A little-known feature in .NET is that there is an event (AppDomain.AssemblyResolve) that fires when it can't find an assembly. By handling this event we can load it manually (for example: from the file system, download-on-demand, or from an embedded resource):

image

This works fine in most applications, but not VS; this event simply never fires within a VS package. Oh well. Another feasible option is to use reflection to load the dll (probably acceptable if the API is very shallow):

 

image

Not pretty, but it'll work - except now you just move the problem downstream; any dlls that our library needs will be missing too, with all the same problems plus some more issues associated with the "load context" (depending also on the whole LoadFile / LoadFrom choice)... oh dear.

Processes vs AppDomains

Our original problem was that files that existed locally weren't being resolved due to the assembly resolution paths of VS; but what if the second assembly was actually an exe? In the case of protogen.exe this was fine. So we could launch a new Process (setting the working path, etc); not elegant, but very effective - in particular it bypasses all the issues of the "load context" inherited from VS; we have our own process, thanks very much.

But there is an alternative... a second AppDomain; this lives within the same process, but is isolated (in terms of assemblies, configuration, etc), and can be unloaded. And neatly, you can execute an assemblies entry-point very easily (conveniently passing arguments as a string[] rather than having to pack them down as a single string for Process):

image

This is actually quite a cute way of launching a sub-process (for .NET code).

Processes within AppDomains within Packages

As it happens, in my case my child dll itself has to launch a Process during execution ("protoc.exe"); normally this is fine, but VS was soon causing problems again; for some curious reason (presumably linked to security evidence), when using ExecuteAssembly this inner Process.Start would always flash a console frame (even when told not to). Oddly, if I instead launch a Process from my Package, and that Process launches a Process, then this doesn't happen. A bit of an undesirable quirk, but as a consequence this multi-level Process tree is currently how the VS package does its work.

But in general you can use ExecuteAssembly as a neat alternative to a Process.

Summary

VS packages have "fun" written all over them... maybe my borked installer was right all along ;-p

Saturday, 25 July 2009

Busy busy busy

Just a quickie...

Apologies if you've sent me a patch, bug, or other piece of work over the last week or so; I will get to it, but things are a bit hectic at the moment; "crunch" time on a project for work, plus lots of non-work things keeping me very busy.

So I'm not ignoring you; I'm just completely maxed. Hopefully normal service will resume shortly ;-p

Friday, 17 July 2009

Things not to do with an installer...

First - important point: this bork never left my laptop; this is not a problem in the package I mentioned yesterday!

Can you spot the face-palm in the following (that installs the protobuf-net Custom Tool into Visual Studio 2008):

image

That will install just fine, and the Custom Tool will work. And then you uninstall it.... oh dear. It helps if you know that {FAE04EC1-301F-11D3-BF4B-00C04F79EFBC} is Visual Studio's nickname for "C#". Did you notice that innocent little "DeleteAtUninstall" flag? Somehow it get set to true (it certainly wasn't deliberate).

Oh well, perhaps I can get by without all the inbuilt tools! Or maybe I need to rebuild my crippled registry... Fortunately this only impacted my laptop before I spotted it (I do test things, honestly!).

Still - it could have been worse; a few levels higher up and I could have completely broken Visual Studio (or potentially windows, but I would hope that the operating system wouldn't let me do something truly stupid, such as deleting HKLM/Software).

The moral of this: when you are writing installers, double check both the install and the uninstall do exactly what you expect. No less, and (perhaps more importantly) no more.