qwtRenderBackground defined twice, breaks unity build

[alex_sh]

Hello,

I'm doing a unity build of Qwt (see http://cheind.wordpress.com/2009/12/...-unity-builds/), and getting huge improvements (30 sec -> 5 sec with make -j1; 7 sec -> 4 sec with make -j9 on my quad-core).
However, there is a duplicate static function that breaks the build: qwtRenderBackground(). It's in src/qwt_legend.cpp AND src/qwt_plot_renderer.cpp. Since both files are included in the unity cpp file, it gives errors.

As a temporary solution, I simply renamed the function in one of the files. I hope the SVN version can be fixed as well (by renaming or by moving it to some header).

Thanks for the great library!
Alexander

[Uwe]

Concatenating all sources into one has got a name - how weird is this.

Maybe there is a very, very specific reason ( someting like a limited compiler on embedded boards ) what justifies this, but otherwise I don't see any good reason for this "unity" build: only Qwt developers ( = me ) compile the library often and I edit/compile single files what is much faster than rebuilding the complete library each time.

In general I won't take care of having unique names for local static declarations but in case of qwtRenderBackground it is a forgotten temporary implementation that needs to be solved in a different way ( maybe as a method of QwtPainter ).

Uwe

[alex_sh]

Thanks for taking care of it!

I understand your reasons for not actively supporting unity builds.

However, please don't dismiss them as being very specific to certain scenarios. We build a local copy of Qwt as part of our project build and since migrating to unity builds (with 20 files per unified cpp) we've seen 3-4x increase in overall compilation speed, and also a nice speed up of incremental compilations as well.
You can configure how many cpp files to unite (15-20 is usually as sweet spot). For a correctly chosen value the difference in speed of recompilation for one original file and one united file is quite minimal (in fact, we don't really notice any). On the other hand, when you say you edit/compile a single file, how often do you also change the header files? If a single header file change causes recompilation of, say, 3 cpp files, then the unity build is already faster. Most of the build slowness comes from invoking the compiler and compiling header files. Unity builds avoid that.
For cmake-based projects a nice framework for unity builds is https://github.com/sakra/cotire (I know you use qmake in Qwt but it may be useful for someone else). Here are also some opinions from OGRE people: http://www.ogre3d.org/forums/viewtopic.php?f=4&t=64175 .
Anyway, I, too, was skeptical at first, but once I tried it - there's no turning back.

Thanks for the great library!
Alexander

[Uwe]

On the other hand, when you say you edit/compile a single file, how often do you also change the header files?

Well my development systems are usually Atom boards ...

The problems are with declarations like "static const int Margin" in qwt_arrowbtn.cpp and qwt_legend_label.cpp. Luckily both are set to 2, but what does your compiler do if they were different values ?

Uwe

[alex_sh]

We don't compile the widgets portion of Qwt, so that particular example doesn't affect us (I imagine the compiler will error out on a name clash even with identical values).

However, you can group the source files (cotire makes it easy to do), e.g. compile the widgets in a separate unity file group, and just make sure you have no conflicts inside these groups.
For name clashes the compiler will always warn or display errors about them. If the code changes are impossible, the problematic cases can also be worked around by manually disabling unity builds for individual files or by splitting them into different unity groups.

For sources that are ours, we tend to do those declarations as static const member variables of classes (sometimes of private implementation classes), or simply prefix them in a way that ensures uniqueness (e.g. qwt_arrowbtn_margin and qwt_legend_label_margin). Hiding all that stuff in private implementation class is usually the way to go.

While it may look like a lot of work, it's actually a lot easier than it looks, just follow some simple rules, and the compiler always says if something is wrong. I was able to create a full unity build of a project (~1400 files, ~250K loc) in just a couple of hours, including setting it all up.

Cheers,
Alexander

[Uwe]

While it may look like a lot of work, it's actually a lot easier than it looks, just follow some simple rules, and the compiler always says if something is wrong. I was able to create a full unity build of a project (~1400 files, ~250K loc) in just a couple of hours, including setting it all up.

The point is that you are doing it with code that is not written with the intention of being concatenated - breaking the scope of the declarations that were intended by the developer ( who knows the code ). And in the end the build times are worse, than what you get from installing Qwt as a library ( because you only have to build it only once - when you install it ).

I know that some projects maintain Qwt in the way you do it. The reasons I heard so far are:

1. private patches
2. being unable to cope with installing and using libraries in general
3. auto updating to SVN

Unfortunately 2 is the main motivation, but what is yours and why do you handle the Qwt source code different as the source code of a Qt module ?

Uwe

[alex_sh]

The point is that you are doing it with code that is not written with the intention of being concatenated - breaking the scope of the declarations that were intended by the developer ( who knows the code ). And in the end the build times are worse, than what you get from installing Qwt as a library ( because you only have to build it only once - when you install it ).

The reasons - well, we have several compilers on several platforms and architectures, so it's a lot easier to just compile it with the rest of the project and not maintain it separately. It's also *a lot* easier to update Qwt from SVN - no need to compile and perform installations on all development computers and in all environments simultaneously. We don't usually have any private patches except for very rare and small fixes that I usually report here.

The build time increase - well, it takes 4 seconds to build Qwt as a shared library on my machine (7 seconds without unity), so it's no big deal, really. And for incremental builds it's already there. Note that I'm not saying Qwt *must* be built in a unity way, definitely not, I'm saying the method is something the developers should try for their own projects - it definitely has benefited us.

Breaking the scope - sure, but I don't see how that would actually break the behavior. If there are name clashes, the compiler reports them and we fix them.

I know that some projects maintain Qwt in the way you do it. The reasons I heard so far are:

1. private patches
2. being unable to cope with installing and using libraries in general
3. auto updating to SVN

Unfortunately 2 is the main motivation, but what is yours and why do you handle the Qwt source code different as the source code of a Qt module ?

I guess all three, for the reasons I mentioned above.

We rarely upgrade Qt, but since we usually depend on the SVN features of Qwt, we update it quite frequently. And, since the SVN version is not quite API/ABI-stable, it would mean that we have to update all the environments simultaneously, which is simply too much work for not much gain. Also, we don't build the widgets part and that reduces the compilation time and the generated library size.

Cheers,
Alexander

[Uwe]

The build time increase - well, it takes 4 seconds to build Qwt as a shared library on my machine (7 seconds without unity), so it's no big deal, really.

What a rocket - on my system it takes several minutes to build the Qwt library from scratch.

Also, we don't build the widgets part and that reduces the compilation time and the generated library size.

For regular builds disabling "QWT_CONFIG += QwtWidgets" in qwtconfig.pri does this.

Anyway I don't promise anything, but at least now I have this scenario in mind. Also using the same name for different functions ( even if static to specific files ) is an accident - nothing I would call good practice. Guess most IDEs ( if I would use one ) would have problems with such a thing.

Uwe

PS: in trunk the 2 qwtRenderBackground functions have been moved to QwtPainter::drawBackground

[alex_sh]

What a rocket - on my system it takes several minutes to build the Qwt library from scratch.

That's an i7-2600 (4 cores, 8 HT-threads), a debug build with gcc on x86-64 linux. The optimized build takes about twice that time.

I just did a quick benchmark:
make -j1: 34 sec.
make -j1 + unity: 7 sec.
make -j9: 8.3 sec.
make -j9 + unity: 4 sec.

(Note: -j9 seems to be the optimal value for this CPU).
The less cores/HT-threads you have, the more the speed benefit of a unity build. On an infinite-core machine I guess the ordinary build would approach the unity one, but until then...

Anyway I don't promise anything, but at least now I have this scenario in mind. Also using the same name for different functions ( even if static to specific files ) is an accident - nothing I would call good practice. Guess most IDEs ( if I would use one ) would have problems with such a thing.

Good enough for me!

PS: in trunk the 2 qwtRenderBackground functions have been moved to QwtPainter::drawBackground

Thanks, I just saw that in SVN.

Take care,
Alexander