Monday, 16 July 2012

Introducing the protobuf-net precompiler

Over the last few posts (here and here) I’ve given a few hints as to compiling for other platforms. Basically, this all relates to how well protobuf-net works on platforms like iOS, WinRT/Metro, Silverlight, Phone 7, etc. These heavily restricted runtimes don’t allow much meta-programming, and they might be running on low-power CPUs, so reflection (even if possible) is not ideal.

I’ve played with assembly generation before, but with mixed results. For example, here for Phone 7. This could just about work for some frameworks, but was patchy on some, and won’t work at all for others.

Well, all that IKVM shininess has opened up a whole new set of tools. The small beauty that I’m ridiculously pleased with is a new utility exe in the SVN trunk (will be part of a proper release soon): precompile.

This unassuming little tool works a bit like “sgen”, but with the ability to target multiple frameworks. What it does is:

  • inspect the input assembly (or assemblies) to resolve (if it can) the target framework
  • initialize an IKVM universe targeting that framework
  • load the core framework libraries, protobuf-net, and the input assemblies into the IKVM universe
  • scan the input assemblies for types marked [ProtoContract]
  • add those to a protobuf-net model
  • compile the model to a type / assembly of your choosing

To use that in a project you might:

  • create a new DTO project in your chosen framework, and compile it
  • execute precompile to generate to a serializer assembly
  • from your application project, reference the DTO and serializer assemblies
  • use the type you created

For example, say I create a new Phone 7 DTO assembly, called PhoneDto (because it is late and I lack imagination). I can then create a serialization assembly via:

precompile {some path}\PhoneDto.dll –o:PhoneSerializer.dll –t:MySerializer

This will generate a library called PhoneSerializer.dll, which you can reference from your main project (in addition to the DTO and the protobuf-net core).

Then, just use MySerializer:

var ser = new MySerializer();
ser.Serialize(output, obj);

I hope this finally solves a number of tooling issues. I’m pretty pleased with it. I’ve tested it against a range of different frameworks, and it has worked well – but if you get problems, just let me know (comment here, or email me, or log an issue on the protobuf-net project site).

43 comments:

allon said...

Could this tool help me precompile a RuntimeTypeModel that is created via code (rather than attributes)?

The RuntimeTypeModel.Compile(string, string) method provides a way to compile a RuntimeTypeModel into a file, but I don't see a straightforward way to do it in the build process (e.g. Post-build event).

If indeed the precompiler only support attribute-based discovery, perhaps you could add another attribute [PrecompileRuntimeTypeModel] that can be placed on a public static method with a return type of RuntimeTypeModel, and the precompiler would invoke that method, compile the model that it returns and include it in the output assembly.

Marc Gravell said...

@allon targetting which framework? If regular .NET, I'd write a console exe that sets up the RuntimeTypeModel and calls Compile(string,string), and call that exe in the build process. If you are targetting another platform, it gets *much* more involved.

allon said...

Yes, vanilla .NET. I just thought the protobuf-net precompiler could be that tool, rather than me (and probably others) having to write one.

Marc Gravell said...

At the moment, the "custom model" approach isn't supported by the precompiler - there's technical reasons behind that.

But by comparison: sgen doesn't let you define XmlAttributeOverrides at runtime *at all*. So having the *ability* to write a console exe *of whatever complexity you like* that builds a model then calls ".Compile(...)" is still pretty cool, IMO.

allon said...

Yep, it sure is!

Felven said...

excuse me, I want a copy of this tool for the development of windows phone 7. could you send to fanzhaoxing@gmail.com ?

Felven said...

excuse me, could you send a copy of the tool to me(fanzhaoxing@gmail.com)? I am busy developing windows phone 7 applications. thanks

Marc Gravell said...

@Felven This is fully available in r565: http://code.google.com/p/protobuf-net/downloads/list

(note: that is a few commits behind, which relate to the portable class library framework; if you're using portable class library, let me know and I'll update the download)

Benjamin said...

This is a precious little gem hidden far too deep in the interwebs, in my opinion.

I am developing an OSM vector map viewer for WP7, and thus need to load the map at runtime on the phone.

Long story short, a map load time of ~35 seconds (DataContractSerializer) has been reduced to approx 2 seconds. Of course the 10x space reduction is nice too.

Many thanks for this! Now I do not have to spend a week on developing my own binary format.

rpnelson said...

Thanks for all your great work, Marc. One question, does the precompiler look for [DataContract] and [DataMember(Order=n)] as well as [ProtoContract]?

tào lao said...

hi Marc,
how do you thing about building a portable version of protobuf-net assembly? which can use both in projects using windows phone, silverlight and server, mean we can now write only 1 assembly for message contracts, ser/deserialize helpers and re-used it in all projects of solution.

Marc Gravell said...

@tào lao the r580 download on google-code already includes portable library builds for both Full are CoreOnly. HOWEVER! I will advise that this has many compromises (due to needing to satisfy all frameworks), and the framework-specific builds are preferred.

Anonymous said...

I have some problem in baseclass and subclass ,i want my dll can use in ios,but when i use attributes(ProtoInclude) there is an error "Cyclic inheritance is not allowed" ,so i need find another way to precompile my dll, i set ProtoMember's "DynamicType= true" and drop ProtoInclude. The dll precompile success. but when i use my dll to Serializer my class there is an runtime error "Unexpected sub-type".Could you give me some help?Thank you,and my english is poor...

Michal said...

Hi,
could you give some example how to use precompile tool on windows phone 7.1 using as input .proto files ?

I need to use .proto, because it allows for same message definition for multiple languages.

Maybe it is obvious, but I am new to .NET and the documentation is 'a bit' lacking in information.

Marc Gravell said...

Starting from .proto, it will be a couple of steps:

0: create a WP7 class libray, called MyDtoLibrary (or whatever you want; the point is: this library is for your DTOs) - and reference protobuf-net from this library
1: use "protogen" to generate C# from the .proto
2: include the generated C# into your library project, MyDtoLibrary
3: build the MyDtoLibrary project
4: point "precompiler" at the newly compiled MyDtoLibrary.dll, outputting something like "MySerializer.dll" (again, do what you like with the names)
5: in your app, reference MyDtoLibrary, protobuf-net, and MySerializer.dll - use the generataed serializer from there (do not use Serializer.Serialize, etc)

nero said...

Excuse me. I found that it cannot precompile an generic object like
[ProtoContract]
public class Result
{
}
in CF 3.5. It catch exception abt
"object reference not set to an instance of an object"

nero said...

should be
public class Result<T>
{
}

Anonymous said...

Hi Marc,

Thanks for posting such an amazing blog.

We are facing huge perfomance issues in my firm due to serializaiton. The problem is serializing between dataset to xml to strongly typed c# object. The solution seems to be an xml serializer which is capable of providing interop between xml and wcf serialization.

> Is protobuf-net capable of doing that?
> Do you know any open source xml serializer?

Please advice.

Thanks

Marc Gravell said...

@anon protobuf uses a defined binary format; it is not compatible with XML. You could try ServiceStack for that, but I can't be sure.

David Hayes said...

How would you go about precompiling a RuntimeTypeModel DLL for (say) WP7. Is it possible (I guess by possible I really mean reasonably feasible)

Marc Gravell said...

@David what do you have at the moment? If you have a self-describing model, the standalone tool will do the job. If your intent is to configure it manually: that is trickier. I could probably prepare an example if you have a concrete example.

David Hayes said...

Hi,
The objects I want to serialize are in code I can't (easily) apply attributes to so I've been using the RunTimeTypeModel. The (not complete) code I have is in this thread OSMSharp.
It's hopefully not essential I precompile it but any performance gains is a good thing

Paul said...

Hi Marc. I post my question here http://stackoverflow.com/questions/17694324/protobuf-in-windows-8-app-serialization-or-generating-code-not-working but nobody response so I want ask you. Do you have any idea where can be problem?

Marc Gravell said...

@Paul I have replied

Christiaan Maks said...

When trying to build the MyDtoLibrary (as per your instructions in an earlier comment), I get the following error when building:
The type or namespace name 'Serializable' does not exist in the namespace 'System' (are you missing an assembly reference?)

on: global::System.Serializable

I thought it might be that I'm referencing the wrong protobuf-net but I've tried several and they give the same error. Any ideas?
(Using VS Express 2012 for Windows 8)

Also, since I'm a little confused because of the directory tree, which is the best to use for WP8? Same question for WinRT. Should I use protobuf-net.dll from the precompile directory for both of these?

Thanks!

Marc Gravell said...

@Christian that sounds more like "protogen" than "precompile" - if you want to target platforms without Serializable, you want the light framework switch - try /? To see the available options.

Christiaan Maks said...

Thanks for the help, I didn't realize there were options :)

I'm getting the following error on the precompile step now though:

D:\Downloads\protobuf-net r666\Precompile>precompile ../ProtoDTOLibCS.dll -o:Awe
someSerializer.dll -t:AwesomeSerializer
protobuf-net pre-compiler
Detected framework: .NETCore\v4.5
Resolved C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCor
e\v4.5\mscorlib.dll
Resolved C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCor
e\v4.5\System.dll
Resolved ..\protobuf-net.dll
Resolved C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCor
e\v4.5\System.Runtime.dll
All assemblies must be resolved explicity; did not resolve: Windows, Version=255
.255.255.255, Culture=neutral, PublicKeyToken=null
at ProtoBuf.Precompile.PreCompileContext.<>c__DisplayClassf.b__a(Obj
ect sender, ResolveEventArgs args) in c:\Dev\protobuf-net\precompile\Program.cs:
line 335
at IKVM.Reflection.Universe.Load(String refname, Assembly requestingAssembly,
Boolean throwOnError)
at IKVM.Reflection.Reader.ModuleReader.ResolveAssemblyRefImpl(Record& rec)
at IKVM.Reflection.Reader.ModuleReader.ResolveAssemblyRef(Int32 index)
at IKVM.Reflection.Reader.ModuleReader.ResolveType(Int32 metadataToken, IGene
ricContext context)
at IKVM.Reflection.Type.get_IsValueType()
at IKVM.Reflection.Type.get_IsClass()
at ProtoBuf.Precompile.PreCompileContext.Execute() in c:\Dev\protobuf-net\pre
compile\Program.cs:line 365
at ProtoBuf.Precompile.Program.Main(String[] args) in c:\Dev\protobuf-net\pre
compile\Program.cs:line 33


I've put the protobuf-net.dll from CoreOnly\netcore45 next to the ProtoDTOLibCS.dll file. Left the precompile folder as is.
I guess its looking for System.Windows.dll ? It is in C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5

Marc Gravell said...

@Christian is there anything unusual in your project? I assume it is just windows-store? I can see if I can repro tomorrow.

Christiaan Maks said...

Just a Windows Store Class Library. Nothing changed other than adding the generated .cs file and referencing the CoreOnly\netcore45 protobuf-net.dll.

The .cs was the lightFramework generated protobuf file.

This was able to build the .dll, which then gives the previous error on the precompile step.

MyLittleDido said...

Hi Marc,

I want to run my Unity3D game on windows phone8, and I have an error when I compile my serializer project under windows 8 platform, but same code works great on both IOS and Android.

------- my code: -------------

var model = TypeModel.Create();
model.Add(typeof(CSLogin),true);
....
model.Compile("mySerializer", "mySerializer.dll");

------ error message: -------
'ProtoBuf.Meta.RuntimeTypeModel' does not contain a definition for 'Compile' and no extension method 'Compile' accepting a first argument of type 'ProtoBuf.Meta.RuntimeTypMode.

------- dev environment: -----
windows 8
visual studio 2012
protobuf-net r666/full/wp8/protobuf-net.dll

Is there something wrong? thanks a lot!

Marc Gravell said...

@MyLittleDido the metro / windows-store / WinRT (whatever you want to call it) build does not include all of the meta-programming layer, due to platform restrictions. This is largely *why* the precompiler (this blog post) exists: the point is - you can create a WinRT DTO assembly (with your contracts), then use the command-line precompile tool, and it will generate a WinRT serialization library for you (basically, it detects the target platform of the input, and acts accordingly)

Simon Labrecque said...

Take a look at Fluent protobuf-net; it allows Fluent-Nhibernate-style type-safe, code only mappings. The mappings can be in a separate assembly if desired.

https://github.com/simon-labrecque/fluentprotobufnet

Pantelis said...

When I have ProtoInclude and inheritance, I get the following error from "precompile":

Value cannot be null.
Parameter name: member
at ProtoBuf.Compiler.CompilerContext.CheckAccessibility(MemberInfo member) in
c:\Dev\protobuf-net\protobuf-net\Compiler\CompilerContext.cs:line 752
at ProtoBuf.Compiler.CompilerContext.EmitCall(MethodInfo method) in c:\Dev\pr
otobuf-net\protobuf-net\Compiler\CompilerContext.cs:line 559
at ProtoBuf.Serializers.TypeSerializer.WriteFieldHandler(CompilerContext ctx,
Type expected, Local loc, CodeLabel handler, CodeLabel continue, IProtoSerializ
er serializer) in c:\Dev\protobuf-net\protobuf-net\Serializers\TypeSerializer.cs
:line 664
at ProtoBuf.Serializers.TypeSerializer.ProtoBuf.Serializers.IProtoSerializer.
EmitRead(CompilerContext ctx, Local valueFrom) in c:\Dev\protobuf-net\protobuf-n
et\Serializers\TypeSerializer.cs:line 583
at ProtoBuf.Meta.RuntimeTypeModel.WriteSerializers(CompilerOptions options, S
tring assemblyName, TypeBuilder type, Int32& index, Boolean& hasInheritance, Ser
ializerPair[]& methodPairs, ILVersion& ilVersion) in c:\Dev\protobuf-net\protobu
f-net\Meta\RuntimeTypeModel.cs:line 1521
at ProtoBuf.Meta.RuntimeTypeModel.Compile(CompilerOptions options) in c:\Dev\
protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 1138
at ProtoBuf.Precompile.PreCompileContext.Execute() in c:\Dev\protobuf-net\pre
compile\Program.cs:line 434
at ProtoBuf.Precompile.Program.Main(String[] args) in c:\Dev\protobuf-net\pre
compile\Program.cs:line 33

Marc Gravell said...

Any model code to go with that? It sounds like something is non-public, but I've missed an opportunity to give a clearer message. With ab example, I could improve that...

JR said...

Hello marc- this is an amazing tool. we are looking to replace xmlserialization, but i am getting hung up trying to use PreCompile. it seems to trip over a "parseable" type. Any work arounds?

...
No serializer defined for type: System.Version
at ProtoBuf.Meta.ValueMember.BuildSerializer() in c:\Dev\protobuf-net\protobuf-net\Meta\ValueMember.cs:line 329
at ProtoBuf.Meta.ValueMember.get_Serializer() in c:\Dev\protobuf-net\protobuf-net\Meta\ValueMember.cs:line 198
at ProtoBuf.Meta.MetaType.BuildSerializer() in c:\Dev\protobuf-net\protobuf-net\Meta\MetaType.cs:line 470
at ProtoBuf.Meta.MetaType.get_Serializer() in c:\Dev\protobuf-net\protobuf-net\Meta\MetaType.cs:line 384
at ProtoBuf.Meta.RuntimeTypeModel.BuildAllSerializers() in c:\Dev\protobuf-net\protobuf-net\Meta\RuntimeTypeModel.cs:line 833
at ProtoBuf.Precompile.PreCompileContext.Execute() in c:\Dev\protobuf-net\precompile\Program.cs:line 398
at ProtoBuf.Precompile.Program.Main(String[] args) in c:\Dev\protobuf-net\precompile\Program.cs:line 33



thanks.

Marc Gravell said...

@JR hmmm; currently you can only do that in code; I will add an option and redeploy

Pantelis said...

I'm trying to port my code WP8 code to Xamarin and Android, and I'm using your precompile to generate a serialization library. It's complaining that it can't resolve mscorlib.dll

C:\Users\PKOUR_000\Downloads\protobuf-net r668\Precompile>precompile "C:\adora\s
chedoole\web.schedoole.com\prototypes\android\Schedoole-Android\Schedoole\bin\De
bug\Schedoole.DLL" -o:Schedoole.Common.Serializer.dll -t:MySerializer
protobuf-net pre-compiler
Detected framework: MonoAndroid\v2.2
mscorlib.dll not found!
Resolved C:\adora\schedoole\web.schedoole.com\prototypes\android\Schedoole-Andro
id\Schedoole\bin\Debug\protobuf-net.dll


I also tried to create a portable class library, but ran into a different set of issues. I can post those if you think that's a better path to follow

Marc Gravell said...

@Pantelis you can use -f to supply the full path to the framework, or in other words: the path to the file system where the mscorlib.dll that your app uses resides

Pantelis said...

I ended up using -f:MonoAndroid/v1.0 and that seems to do the trick

animalnots said...

Is update for serializable model supported? For example I'm using precompiled table structure for windows phone 7 app with 5 columns with a lot of data filled. Perhaps, I want to add or remove a column (a field) without loosing data. What's the best way to do it? Is there any mechanism like described here:
https://developers.google.com/protocol-buffers/docs/proto#updating
like: "Any new fields that you add should be optional or repeated. This means that any messages serialized by code using your "old" message format can be parsed by your new generated code, as they won't be missing any required elements. "
or "Non-required fields can be removed, as long as the tag number is not used again in your updated message type (it may be better to rename the field instead, perhaps adding the prefix "OBSOLETE_", so that future users of your .proto can't accidentally reuse the number)."

Marc Gravell said...

@animalnots sure, just do what it says! add or remove a member, keeping in mind not to use the tag-number again (bad things will happen with old data etc). Did you hit a problem? if so: what?

deepahnika said...

Hi Marc,
I am getting the following error when I use protogen to generate the c# model. The protobuf-net.dll I use is the core-only ios version.

protobuf-net:protogen - code generator for .proto
Method not found: '!!0 ProtoBuf.Serializer.Merge(System.IO.Stream, !!0)'.

Dzmitry Lahoda said...

Small convenience for users:

if
-f:.NETFramework\v3.5\Profile\Client

replaced with nuget ids:

NET35, wp7, sl5, MonoAndroid.

And put precompiler onto nuget either.

http://docs.nuget.org/docs/creating-packages/creating-and-publishing-a-package

https://nuget.codeplex.com/workitem/2847