While working on 10.2 updates, we started delving into some performance issues of the Delphi RTL, that were effecting other libraries like JSON processing, streaming, string processing, DataSnap and RAD Server web services, and more. In many cases, addressing these issues required adding new methods to existing classes, or adding new global functions, something we can do only in interface breaking releases like 10.3.
Optimization and performance was a guiding factor to many of the changes listed below, but not the only element. We also wanted to update external libraries, improve conformance to standards like JSON and HTTP, and improve quality overall. With this scenario in mind, the list of improvements is fairly long (and a bit boring, sorry).
Table of Contents
Change in Data Structures Growth Strategy
Several data structures (like TStringList, TList, TList <T>, TQueue<T> and TStack<T>) have now a flexible growth strategy when the internal storage is full and needs to be expanded after adding an additional item. In the past, the strategy was to double the size of the underlying structure, something that works fine for smaller sizes, but not when you have several MB of storage. Now the rules is to increase capacity by 1.5 times. For example you can check a TStringList growth by checking the Capacity property, which now grows as follows (from 14K to 74K):
14,761
22,141
33,211
49,816
74,724
This behavior, shared by multiple collection classes, is defined in the new function, declared in SysUtils.pas:
function GrowCollection(OldCapacity, NewCount: Integer): Integer;
Now the interesting element is that the implementation can be replaced with a custom function growth strategy, by writing a new compatible function and calling the global SetGrowCollectionFunc procedure.
Changes in TStringBuilder Class
The TStringBuilder class has had several changes with the goal of improving its performance, including a similar change in memory growth strategy, the removal of some redundant code, and an overall implementation cleanup. The TStringBuilder enumerator has been optimized, however it expects that the TStringBuilder object will be not modified while the enumerator is processing it (which is a common behavior for enumerations).
There is also an additional parameter for the TStringBuilder.ToString method. The signature is ToString (UpdateCapacity: Boolean). ToString(True) will give better performance if no more modifications expected for TStringBuilder, as it reduces the amount of data being copied.
Lists-Related Improvements
There are other improvements related with lists and collections. The generic TList and the generic TDictionary have new public properties to make their comparers (the definition of their comparison operations for sorting) accessible after initialization. We’ve added a TryAdd method to TDictionary, an ExtractAt to TObjectList and significantly improved the performance of several operations (IndexOf, Add, and more). We have also specifically optimized for-in loops for generic collections and string lists, with an empty “for in” loop about 3 times faster
JSON-Related Improvements
In 10.3 we have improved the correctness of JSON content, in terms of the JSON code generated by the TJSONValue class and derived ones, but also in terms of parsing. We have also worked on performance improvements. There is a new TAsciiStreamWriter class that can be combined with a TJSONTextWriter to give the best JSON string generation performance (as it does almost no conversion at all). There is now “pretty print” JSON output with the introduction of the new TJSONAncestor.Format(Indentation: Integer = 4). As a consequence, TJSON.Format has been deprecated.
We have also clarified (and properly implemented) the difference between calling ToJSON and ToString for a JSON object:
- TJSONAncestor.ToJSON always produces a formally valid JSON string
- TJSONAncestor.ToString produces a similar JSON string, but without converting non-ASCII symbols to \uNNNN (faster, but not formally valid)
JSON parsing support has a new behavior in case of errors in the JSON source text, allowing you to raise an exception with information about the error position in the source text, rather than just returning nil. The new option controlling the behavior is TJSONParseOption.RaiseExc. If the exception is enabled, ParseJSONValue raises the new System.JSON.EJSONParseException (which has the properties Path, Offset, Line, and Position). Additionally, the method TJSONObject.ParseJSONValue has a third new parameter: RaiseExc, which overrides the global setting causing the exception to be raised, in case of JSON parsing errors.
TMemIniFile Class Improvements
In Delphi 10.3 we have optimized the TMemIniFile implementation. Reading and constructing a TMemIniFile is 10 to 25 times faster and consumes half of the memory. Other TMemIniFile operations are improved too and they are 50 to 100% faster compared to the previous implementation.
We have also added the ability to load a TMemIniFile from a stream, with two additional overloaded constructors: TMemIniFile.Create(Stream) and TMemIniFile.Create(Stream, UseLocale). These constructors parameters remain available in the class and are exposed in new properties, Stream and UseLocale.
External Libraries Updates (zLib, PCRE, Unicode)
We have done some improvements to external libraries we incorporate:
- zlib has been upgraded to version 1.2.8 with additional fixes (and it is now compiled with RAD Studio C++ compiler for 64-bit)
- The regular expression engine, PCRE, has been upgraded to 8.42 (and also now compiled with RAD Studio C++ compiler), and supports the UTF-16 native version on Windows (to reduce the conversions from UTF-16 string to UTF-8 strings).
- The Unicode table (System.Character unit) supports Unicode v11.0.
A Lot Was Done in 10.3 for Delphi RTL
As you can see the number of improvements in Delphi RTL for 10.3 Rio is fairly significant, not to mention the work focused on fixing bugs and improving quality. There is more for the HTTP and other client libraries, which I’ll cover in a separate blog post.