So I found the answer today. At least under Linux, always a debug build is generated, which is reasonable. But when you 'make install' your project, a 'strip' command is run on the executables, so that you have the release builds in the final folders.
In the end the Qt5 executalble is approximately 20% smaller in size than the Qt4 one.