When in Rome, do as the Romans do - Part II

-

In the previous article, we learned that native Delphi serialization is based around published properties, and that other kinds of serialization, like field-based serialization, don't fit well with the Delphi model.

Photo: Gary Todd from Xinzheng, China, PDM-owner, via Wikimedia Commons, https://commons.wikimedia.org/wiki/File:Colosseum_(48412957912).jpg

We have also learned that property-based serialization has some limitations and that it does not have out-of-the-box support for the serialization of records, arrays, or indexed properties, and you can only serialize collections defined as TCollection, whose items are descendants of TCollectionItem.

While serialization as a concept is widely used to convert any kind of objects into various storage formats, the main purpose of the Delphi serialization model being built on top of published properties and descendants of TComponent was to serialize the user interface and data modules. This is where the original type limitations came from. Simply put, they supported the types needed to serialize visual controls and other design-time components. Since those types also had to be supported by the IDE and property editors, having a single supported collection base type in the form of TCollection was a sensible choice at the time.

But with the provided basic building blocks and RTTI functionality emitted for published properties, you could write your own serialization frameworks with extended support for other types and the ability to serialize your objects to various formats. But before the introduction of the enhanced Delphi RTTI in Delphi 2010, all that was done using published properties.

Enhanced RTTI and its ability to access information about the whole object, rather than just the published section, along with broader support for various types, opened up the possibility of serializing fields (even private ones), not only published properties.

And this is where the story of TList<T> begins...

Around that time, the Delphi language got plenty of new features. Besides enhanced RTTI, there were anonymous methods, generics, attributes... A few years later, Delphi stepped out into the mobile world, adding support for Android and iOS platforms.

And with all that, support for another rapidly growing technology—REST services—found its way into the core Delphi libraries and along with it, the need to provide JSON serialization for simple objects that commonly contain lists and arrays.

But the built-in JSON serialization library went in the wrong direction from an architectural point of view. Instead of using published properties for serialization (maybe due to a lack of out-of-the-box support for arrays and lists), it uses fields. Of course, the ability to serialize arrays and lists fields is something that had to be written from scratch, but there is nothing that would have prevented writing similar code that works on properties. In other words, there is no technical reason why fields had to be used over published properties.

The first issue with field-based serialization is that unlike properties, which usually have an actual name you want to serialize (with the possible exception of casing, that can be auto-converted), fields in Delphi classes are usually prefixed with an F. But field names are really an implementation detail, not a part of the public API in any way, so some developers were using other prefixes for fields.

Suddenly, with the introduction of REST service support, what used to be an irrelevant internal detail, now became part of the public API. And the first roadblock with using the built-in JSON serialization was an issue with fields that were not following the F prefix convention. Yes, I know you can change the field names, but this is just another alarm showing that field-based serialization imposed on existing code is not the right approach—not the Delphian approach.

In the previous article, I had already mentioned "garbage" fields that would end up being serialized by default, and the framework-dependent way to get rid of them that required adding custom framework-related attributes.

But, what all that has to do with the story of TList<T>?

Well, JSON serialization of TList<T> is probably the most broken feature in the history of Delphi. It would routinely get broken between Delphi versions because TList<T> was going through some heavy internal reorganization in order to increase performance and reduce code bloat with almost every new version. And it had some extra garbage to begin with. Since TList<T> is one of the core classes, adding any serialization framework dependency to fix and define its serialization would be a huge red flag.

Some of the bug reports:

And even if TList<T> received some kind of special treatment in REST.JSON serialization, that still would not solve the issues and provide uniform serialization of classes that use custom collections. Basically, instead of using feature-rich lists, you would have to use plain dynamic arrays in order to get consistently serialized fields.

And how does this look today, in Delphi 10.4 Sydney?

  TFoo = class(TPersistent)
  protected
    FData: string;
    FTemp: string;
    FItems: TList<Integer>;
  public
    ...
  end;

var
  Foo: TFoo;
  Str: string;
begin
  Foo := TFoo.Create;
  Foo.Data := 'abc';
  Foo.Temp := 'temporary';
  Foo.Items.Add(4);
  Foo.Items.Add(5);
  Foo.Items.Add(6);
  Str := TJson.ObjectToJsonString(Foo);
end;

Extending the class from the previous article with an integer list, and serializing it with the default JSON options we get a nice, neat... uh, oh.... JSON string

{
  "data": "abc",
  "temp": "temporary",
  "items": {
    "listHelper": [
      4,
      5,
      6
    ]
  }
}

At the end of the day... if you need or want to use the built-in JSON serialization (and if you use DataSnap, you will have to use it) the only way to save yourself in the long run is to carefully handcraft your classes, and that usually means limiting their use and reuse as business objects, which will then lead to unnecessary code duplication.

Conclusion

Well, this started as a more general article advising the usage of coding patterns that fit the language and avoiding patterns that aren't appropriate in it. That advice still stands: No matter what kind of code you write, stick to what's most appropriate for Delphi. That does not mean that other languages don't offer easily transferable and useful coding patterns, though. Just think a bit before you apply them.

As far as serialization is concerned, there are two separate issues here, and both should be considered when choosing any kind of serialization framework.

The first is using field-based serialization instead of published properties. Now, if that could be easily configured, then it would not matter so much, as developers could choose their preferred behavior based on their existing classes.

The second, and the most important issue, is the inability to configure serialization without adding a serialization framework as a dependency to your business objects. That is a huge, enormous red flag. Any attribute-based serialization is pure evil. You should avoid it like the plague. A good serialization framework will allow you to configure everything without requiring you to add it as a dependency. Also, a good serialization framework will use the best defaults, so that the configuration code could be as simple as possible in most cases, while still allowing everything to be tweaked when needed.

And last, but not least, a good serialization framework would not even offer attribute-based serialization configurations, because it is just unnecessary code for a really bad serialization approach. A good serialization framework would not trick its users into using such code, ever.

Of course, you may find a framework that satisfies the first two conditions, but not the last one. If you cannot find any better solution, then the first two are good enough. But for the love of your favorite deity, just don't use attributes for configuring serialization, unless you absolutely don't have any other choice.

Comments

Post a Comment

Popular posts from this blog

Coming in Delphi 12: Disabled Floating-Point Exceptions

-
Delphi 12 is in the works. One of its small features that will have a huge impact is a change in the default handling of floating-point exceptions, which will now be masked on all platforms. Previously, floating-point exception handling was different across platforms, and most notably on the Windows platform: In VCL and console applications, floating-point exceptions were not masked and would raise an exception. On the other hand, FireMonkey applications had all floating-exceptions masked by default, regardless of which platform they were running on. Basically if you haven't explicitly changed the exception mask in your code to some value other than the default, and you had code that attempted to divide some number with zero, you would get an EZeroDivide exception in previous Delphi versions on the Windows VCL application. In Delphi 12, there is no exception, and the result of such a division will be +INF . So the following code will no longer raise an exception. If you were r
Read more

Beware of loops and tasks

-
Loops and anonymous methods don't go hand in hand. Any variable that changes for each iteration will cause problems if it is used within an anonymous method. I explained the mechanism behind variable capturing in the post Mysterious Case Of Wrong Value , along with the problem it causes with loops and its solution. Commonly, anonymous methods are used as callbacks or event handlers, where there are no loops in the context where variable capture happens. While there is a possibility to have an unexpected—wrong—value in such scenarios, it is rather easy to debug them and understand the order in which some code is called, and why the captured value is wrong in a given place. With the rise of multi-threading usage, and especially the Parallel Programming Library and tasks, writing anonymous methods inside a loop becomes more common, and it is much easier to write code where a captured variable will have the wrong value. The biggest problem with such code is that it will be harder to
Read more

Catch Me If You Can - Part II

-
It has been quite some time since I wrote the Catch Me If You Can post about the inability of LLVM-backed compilers to catch non-call hardware exceptions. In other words, hardware exceptions raised by code written in the same block as the try...except exception handler. Since then, more LLVM-backed compilers have been added to Delphi, and with those, more reasons to change common coding practices... but still, very few developers know about the issue. The original article covered only try...except exception handlers, and exploring what exactly happens with implicit and explicit try...finally handlers was left as an exercise to the readers and to some future, now long overdue, post. And Now: All Hell Breaks Loose When you read about non-call hardware exceptions not being caught by an immediate try...except block, the implications might not look too serious. After all, while plenty of code can raise exceptions, try...except handlers are not that frequently used in places, as mo
Read more