Tuesday, 10 November 2009

Controlling WCF / protobuf-net at the Endpoint

protobuf-net has had WCF hooks for a long time, but (to be frank) they’ve been a tad…. flakey.

Firstly, if you use “mex” generated proxy classes, it has a habit of discarding key metadata (the “Order” of the [DataMember]). This can be resolved either by using a shared DTO assembly, or by using fixups in the partial class – but…

Secondly, even if you have a shared DTO assembly, if you use the standard “add service reference” (or svcutil.exe), it duplicates the service-contract interface, and it drops the [ProtoBehavior] marker. Again, you can get around this by using the WCF channel directly or subclassing ClientBase<T> – but it would be nice if this wasn’t necessary.

Additionally, it would be very nice if the protobuf-net aspect was configurable (app.config/web.config); allowing greater reuse, and allowing parallel endpoints (with/without protobuf-net, to support vanilla and protobuf-net aware clients).

The community to the rescue

Step up Scott Prugh, who an embarrassingly long time ago very kindly supplied me with all the cogs to make this happen. Apologies for the delay, but things have been hectic and when I tried to test it my WCF brain-cells were switched off (in short: I borked my tests, couldn’t get it working, and had to shelve it for a while).

So what’s the shiny? Simply: rather than decorating your operations with [ProtoBehavior], we can apply it in the configuration instead, by adding an endpoint-behavior that references a custom WCF extension (which we must also add). This sounds worse than it is'; inside “system.ServiceModel” (in your app.config/web.config, this means adding a few lines of boilerplate:


And then to both client and server, we simply specify the endpoint configuration:


(in more complex scenarios you might want to add the custom extension to your existing endpoint configuration).

What’s the catch?

In the most part, that’s it. WCF will use protobuf-net for any suitable objects (data-contracts etc). Note that this is a coarser brush than the per-operation control, though (you could always split the interface into different endpoints, of course).

Also, protobuf-net does have some subtle differences (especially regarding empty objects), so run your unit tests etc.

Note that it only works on the full-fat WCF; it won’t help Silverlight etc, since it lacks the extension features – but that isn’t new here.

Finally, the resolver in WCF is a pain, and AFAIK wants the full assembly details including version number; so one more thing to maintain when you get new versions. If anyone knows how to get around this?


A minimal (but working) example is in the repo (client, server). This uses a shared DTO assembly (to avoid the need for fixups) and is based on the WCF project template. I’ll try to go back and retrofit the Northwind example shortly, and perhaps include the WCF samples in a simple download.


Mikhail_T said...

Hi I tried to play with the sample and seems that it works only if I am using shared DTO object if I using generated proxy objects it fails

Marc Gravell said...

That is explained in the blog; when using proxy objects "mex" tends to drop the all-important "Order" (from [DataMember]). *if* you have to use the "mex" approach, you need to give it a clue how. For now, that means adding a partial class (per-entity) with (at the class level) [ProtoPartialMember(123,"abc")] for each member, where 123 is the tag and "abc" is the member-name. In short, this isn't pretty. I'd like to add something tidier - a DSL or similar. But that is the current state.

Marc Gravell said...

"tag" is synonymous with "Order" for the purposes of the previous comment; in either case; you can of course also use auto-tagging (by name) by enabling that with (again in a partial class, at the class level) [ProtoContract] with a combination of ImplicitFields and InferTagFromName.

Mikhail_T said...

Thank you for the answer. Looks like add service reference generates code in very strange way
I extended you class with some additional properties for test and on this additional properties order property was preservered on the attributes

Jader Dias said...
This comment has been removed by the author.
Jader Dias said...

I was testing the time a WCF service would take to receive a large amount of data, and I didn't notice any improvement when using the protoEndpointBehavior.
Since then I think that serialization isn't a performance bootleneck in WCF services performance.
If you disagree with my results you could point me a benchmark that used WCF services with your protobuf-net versus without it (I already know about your benchmark but I don't think it tested WCF services).

Marc Gravell said...

It all depends on the scenario - if you have two nearby machines making lots of small calls, then it probably won't help. For machines with bandwidth as a key factor (i.e. internet), it really can. It is a bit hard to respond in much more detail without a reproducible example, but network tracing (and more simply, timings) have shown a significant reduction in network traffic when used appropriately.

The other question might be: are you exchanging suitable data?

Marc Gravell said...

For examples:
http://code.google.com/p/protobuf-net/source/browse/#svn/trunk/WcfPerfTest (which uses method attributes), and http://code.google.com/p/protobuf-net/source/browse/#svn/trunk/TestWcfClient | Server | Dto, which uses (IIRC) config.

Jader Dias said...

Thanks for the answers Marc.

You're right.

Yes I didn't test through a narrow bandwidth.

I'll check if my data is suitable.

Jader Dias said...

I noticed that besides of the inclusion of the protoEndpointBehavior, protobuf behaviorExtension and the protobuf-net.dll my client still communicate with a server that don't use protobuf at all.

It is a sign that my client isn't using protobuf-net for serialization.

What have I missed?

Jader Dias said...

I think I got the problem. I was thinking that I could include protobuf-net in my WCF with no recompiling. But it seems to require a [ProtoBehavior] Attribute in the contract.

Marc Gravell said...

If protobuf-net was hooking into WCF, then yes: it would throw a major tantrum talking to a regular server. It is very hard to diagnose what you might have missed remotely, but:
- added the custom extension? (config)
- told the binding to use the extension? (config)
- refreshed the server?

brent said...

Nice blog entry about WCF and protobuf-net.
I have a situation actually, i have a server/client system (socket programming) using protobuf-net as serializer and deserialzer of objects sent through the wire. Im planning to have my server be compatible across other platforms. Im just wondering if it is possible to share the data definition through WCF and use it for the socket programming (protobuf-net codec). With this blog entry, will this also apply that auto generated code at the client side is ready for protobuf serialization at the socket side of the code?.. Would be very glad to hear your insights about this situation.. :)

Marc Gravell said...

@brent - mex is... not 100%; it works best with a shared DTO (either class or binary works) at both ends. I'd be open to suggestions for a better protocol than WCF/mex, but for socket etc I would suggest drop WCF completely and just focus on what is needed.

brent said...

thank you for your quick answer.. as much as i would like to stick to the current implementation(no wcf, shared dto), ill try another way to enable the server communicate with another platform. Thank you so much. Have a nice day.

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

please delete the comment above: Blogger/Blogspot removed all xml elements of the post -- and meanwhile I noticed that VisualStudio magically had duplicated the client-endpoint-element.

Marc Gravell said...

@anon - ok; I assume it was simply a configuration problem, now resolved?

Anonymous said...

Yeah, "update service reference" messed up the client side app.config.

But now I'm stuck with EntityFramework entities: Although the operation contracts clearly expect the entity type, protobuf complains about the base type "StructuralObject" missing a ProtoIncludeAttribute -- but I have no interest in the basetype (which I cannot change).

Marc Gravell said...

@anon - hmmmm; steps to repro?

sakthi said...

Hi iam gonna use protobuf-net in my silverlight application and i understand that i can't make use of behaviors and extensions in app.config, so i have to use the default [ProtoBehavior] attribute for the specific operation method in the interface. Can you please let me know how to configure the same(ProtoBehavior) in the client(silverlight) application

john son said...

Thanks for your wonderful post.It is really very helpful for us and I have gathered some important information from this blog.If anyone wants to get Dot Net Training in Chennai reach FITA, rated as No.1 Dot Net Training Institute in Chennai.

Harshita said...

This is exactly what I was searching for. Awesome post. Thanks a bunch. Helped me in taking class for my students. Wish to follow your posts, keep writing! God Bless!
Dot Net course in Chennai | Dot Net course in Chennai | Dot Net course in Chennai

Anonymous said...

Hi, I am going to use ProtoBufFormatter in my web API application. It is serializing the response, but I want to consume in my JQuery Ajax call function, is there any java script library available to de-serialize or get back to original value?
Thanks in advanced

Shashaa Tirupati said...

Thanks for the article. It has given me so much information that I shared with my students in the class. Come up with more such articles.
Dot Net training Chennai | Dot Net training Chennai | Dot Net training Chennai

toebens said...

how to configure these behaviors in code and not use the .config? how does it look like for server and client configuration in code?

Ashwini Suresh said...

Hello admin, thank you for your informative post on hadoop training in Chennai. It helped a lot in training my students during our hadoop training Chennai sessions. We at Fita, provide big data training in Chennai for students who are interested in choosing a career in big data.

for IT the said...

Great Article

WCF Training | WCF Online Training | Online WCF Training | WCF Training in chennai | Dot Net Training in Chennai | .Net Online Training

Anonymous said...

It is very powerfull when we tried standard WCF that processing a list players for our fifa ultimate team autobuyer (http://futbotmanager.com), it request a lot of time and resources to process very big list of items.
With protobuf everything work fine.

futbotmanager said...

It is very powerfull when we tried standard WCF that processing a list players for our fifa ultimate team autobuyer (http://futbotmanager.com), it request a lot of time and resources to process very big list of items.
With protobuf everything work fine, we reduced around 60% of cpu use.