Offering some tips and suggestions you can consider to help the compiler be faster and consume less memory using the current version of Delphi.
Over the last couple of years, Embarcadero has put a significant effort into optimizing the Delphi Object Pascal compilers’ performance and reducing its memory usage, so that our customers with large applications can have a better experience working with Delphi. The primary focus has been on the Win32 compiler, but other compilers have been improved as well.
We made significant progress in many use cases scenarios, but at the same time by looking at many projects from our customers we discovered a number of cases in which the compiler isn’t behaving as expected. We plan addressing most of those problems in the near future, even if some of the scenarios are bordering on being invalid (like passing non-existing folders to the compiler). At the same time we are getting requests from customers if there is anything they can do in their project configuration or source code to make things better for their developer today.
This is the reason for this blog post: Offering some tips and suggestions you can consider (if they are applicable to your code base) to help the compiler be faster and consume less memory using the current version of Delphi.
Notice that most of the ideas mentioned below help both with compilation and when using Code Insight powered by Delphi LSP, given it uses the same compiler behind the scenes. By helping the compiler, you are also helping Code Insight.
Table of Contents
Invalid Search Path or Library Path Entries
The compiler looks for any unit referenced in the code by uses statements in the project folder, the project search path, the library path, and some other locations. We have noticed that having invalid folders in these paths causes the compiler to repeatedly look into those locations, an operation that is much slower, at the operating system level, for non-existing paths.
You can see an example of a report we received, titled ‘Delphi compiler works extremely slowly when several invalid paths are present in LIB PATH’ at https://quality.embarcadero.com/browse/RSP-3931, as an example.
We made some tests on small applications and for the following for a test case:
- Compilation time with invalid path: 40.3 seconds
- Compilation time without invalid path: 1.7 seconds
As you can see, the difference is huge. The plan is to change the compiler to verify and exclude invalid paths upfront, but it shouldn’t be too difficult for developers to clean up that list themselves.
Source Code Files on UNC drives
A related issue is that some file access operations (specifically looking for missing files) are significantly more expensive when compiling a project entirely or with units on a network mapped drive, compared to using a local drive.
Actually, we fully recommend keeping your source code on an local SSD drive: Delphi compilations are quite disk intensive and a fast hard disk can make a huge difference.
Unit aliases offer the option to map a unit name in a uses statement to a different real unit name. This can be handy when migrating old code and was used by Delphi over the years, when some of the core units were merged or renamed.
For example, in some early versions of Delphi you’d find the following unit aliases:
Today, there is no default unit alias defined for new projects, but you might have existing projects with unit aliases. The recommendation is to remove them (possibly one by one) and update the source code of your projects with references to the correct units.
A large use of unit aliases adds some extra work to the compiler. While to our knowledge this causes only a very little extra overhead, it is still a bit of extra time and the effort also helps you keep your code cleaner and more readable.
Unit Scope Names and Unqualified Unit Names
Over ten years ago, Delphi introduced unit scope names, that is a prefix in front of all units that are part of the default libraries, like Vcl.Forms or System.SysUtils. This is covered in https://docwiki.embarcadero.com/RADStudio/Alexandria/en/Unit_Scope_Names
If you have existing projects with short unit names, a fast migration path is to define unit scope names as part of the project options, as you can see here:
As you can see, the Delphi IDE still adds a large number of unit scope names by default even to new projects (this is something we should consider changing, but that’s a separate point). Now unqualified unit names in your code cause a file look up for each of the possible prefixes, in all of the folders of your search or library path. Consider having 20 folders and 20 prefixes, that’s potentially 400 file locations to look for.
No surprise we have reports like this one, titled ‘Unqualified unit names cause massive amounts of file lookups’ and logged as https://quality.embarcadero.com/browse/RSP-18130. The compiler has some caching logic to reduce the work, but again, this is only a feature to help code migration, but developers should eventually clean up their uses statement, move to use fully qualified unit names for all units in Embarcadero libraries and remove all unit scope names configurations at the project settings level. This is another code change that can have a significant benefit on the compiler.
Repeated and Duplicate Unit Names
Speaking of unit names, there is another scenario that is known to cause trouble. In this case it is less of a performance issue, but a scenario that can cause failures, particularly in Code Insight and some of our IDE tooling.
The scenario is the following: you have a project group open in the IDE with multiple projects including different units (in different folders) with the same name. In other words, one project has MyUnit.pas in a folder and the other project has a different and unrelated unit also called MyUnit.pas in a different folder.
This is a scenario the compiler generally handles fine, but not always as you see in the report ‘Compiler confuses unit names in project group’ at https://quality.embarcadero.com/browse/RSP-39293. We did specific work a couple of years ago so that the debugger won’t get confused in similar scenarios. However, DelphiLSP can get confused by units with the same name.
Whenever possible, it would be much better to avoid using the same name for different units.
Circular Unit References
Getting back to scenarios that affect the compiler performance, one of the most critical ones we know of is the excessive use of circular unit references in the implementation section of the units. As you know, the interface section of the units does not allow mutual or circular unit references. However, in the implementation section of a unit, you can theoretically refer to all other units in the entire project.
When Delphi compiles a unit, it has to look at all used units and navigate the entire graph, making sure of source to check the cycles — to avoid an infinite loop while searching. In the case of a complex graph with hundreds of units the cost of this navigation process is significant.
Recently our support team made an interesting experiment. They created a code generator that creates an application in which every unit uses every other unit. This is clearly an extreme use case — I really hope you don’t have projects like this! In such a scenario, adding one more unit almost doubles the compilation time. In other words, there is an exponential degradation of the time to compile when one more unit is added that uses all other units in the project and is used by all other units (it is exponential as every additional unit adds n*2 unit references where n is the number of existing units).
Again, this is a scenario the compiler was already optimized for, moving searches and name lookup to used hash dictionaries and similar optimized data structures, and an area we are actively looking right now to improve to compiler. See for example this report, resolved in Delphi 10.4.2: https://quality.embarcadero.com/browse/RSP-28811
However, the use of circular uses statements is also a use case scenario that goes against any good programming practice, as you end up with a spaghetti graph of used units. Reducing the dependency across units and across classes is a good practice. Having some linear dependencies (low level units used by high level units, UI units using data access units and not vice versa) is acceptable, but having a complex graph is not healthy for the maintainability of your code over time. In addition, it does put some significant stress on the compiler performance and memory usage.
We do have actual reports from Delphi developers with fairly large applications that reorganizing their uses statements to avoid circular unit references cut up to 90% of their compilation time, like ten minutes getting down to one minute and also helped Code Insight become responsive.
In case you want to address this issue, I recommend considering one of the tools below, which can help you determine the current uses units graph of your application and detect cycles:
- GExpert Project Dependencies (see image below): https://gexperts.org/tour/index.html?project_dependencies.html
- Peganza Pascal Analyzer: https://www.peganza.com/#PAL
- Delphi Unit Dependency Scanner: https://github.com/norgepaul/DUDS
As an aside, notice that the compiler also dislikes when the same units are used in a different sequence (as this affects the dependency graph of units using those units). In other words, if two units both use unit A and unit B, writing “uses A,B;” in both cases is better than having “uses B, A;” in the second one. This is fairly minor, compared to avoiding circular unit references, but still noticeable in a very large application.
Finally, you could also consider removing unnecessary uses statements, one tool offering specific help on this is Delphi Parser:
Repeated Generic Types Instances
The last suggestion we want to offer in terms of helping the Delphi compiler is a bit more contentious and something the compiler should be better at, after a few years. However, we know this can make a dramatic difference, in this case both in terms of compilation time and memory usages, avoiding the risk of out-of-memory compiler errors.
Delphi type system (like the classic Turbo Pascal one) is based on type declaration equivalence. In other words, even if two types have the same name and the same structure, they are different if declared in different units. If you assign a variable of type MyUnit1.TMyType to a variable of type MyUnit2.TMyType, you get a compiler error, regardless of the fact the two types have the same identical structure.
The only exception to this rule is for generic types. If you declare a variable of type TList<string> you are also creating an implicit type, TList<string>, based on the generic type TList<T>. You can have multiple type instances declared in different units and all matching the TList<string> signature and the compiler considers them compatible and allows you to assign them one to the other.
What happens behind the scenes, though, is that the compiler generates a new temporary type of each of the occurrences in different units. So you’ll have, say, TList<string> in MyUnt1 (with a temporary type name) and TList<string> in MyUnit2 (with a different temporary name). For each of these types, the compiler creates instances for the generic type but also for each of the generic methods of the generic type. With a large amount of complex generic types (inherited and nested), this can take a significant effort by the compiler.
The solution to this issue is not to stop using generics, but to create shared instances of generic types whenever possible. In other words, rather than using TList<string> is dozens of different units, you can declare
TListOfStrings = TList<string>;
and use the TListOfStrings type all over the places where you’d use TList<string>. In simple examples, the difference is barely noticeable. In a large application with significant use of generic types, it can make a very significant difference.
Again, we all know that the compiler could handle similar scenarios better, and we have been working to improve this use case — and we did improve it quite a bit over recent years. Still, you can help the compiler be faster and use less memory by refactoring some of your code in that direction — that is writing code like the generic type equivalence didn’t exist.
Is There More? Possibly, Yes
These are some of the scenarios we have found when examining the Delphi compiler performance that can be partially addressed by changes to the code. I’d particularly recommend removing unit aliases and unqualified units, along with the detection and removal of circular unit references. If you are making significant use of generics, consider also the last suggestion above.
This is recommended in particular if you have a large application and you are experiencing slowness in the compiler: Most of the Delphi developers are quite happy with the compiler performance and won’t need to do anything in this respect.
I’m also quite certain I didn’t list all of the scenarios that cause a compiler slow down that can be partially addressed by code or configuration changes. If you have additional suggestions, based on your experience, they are more than welcome!
PS. Have a nice holidays season and all the best for 2023.
Design. Code. Compile. Deploy.
Start Free Trial Upgrade Today
Free Delphi Community Edition Free C++Builder Community Edition
Very interesting article -> congratulations Marco !
I don’t know where to post it. But this is my wish list for now:
1. Compiler directives to force certain parts to be Case Sensitive
2. Splitting of forms into several DFM files (so that a team can work on same project)
I tried the evaluation version of “Remove Superfluous Units” tool by DelphiParser suggested by Marco, and it succeeded in finding 40 unneeded units in my code. However, it didn’t fix the code or list the needed changes, so there’s no way to validate the results without paying $975 for the full version. I solved that problem by using the scroll capture tool in CaptureWiz to copy the tool’s lengthy output log. Then I could page through the log and implement each change myself. I’m familiar with CaptureWiz because I wrote it.
As usual, great article. Congratulations Marco. However, the article focused solely on compile time. Therefore, I would like to ask how and/or how much the mentioned problems affect the memory usage? I’m asking because we have a huge problem with the frequent “Out of memory” error that occurs very often during the build, and more specifically during the linking phase. We are using Delphi XE4 so that could be the cause of the problem too.
Some of the same optimizations affect memory usage. The most relevant one for memory is the duplicate generics data types, as those are distinct types in memory and linker has to figure them out