QSslSocket memory leak

[mentalmushroom]

I asked how much RAM you had in your computer, not how much you saw allocated to your process.

On the test server there is 512 Mb. I thought you mean why am I so worried about 60 Mb. I just tried to say memory grows significantly if I test it well, so this is why I care about it.

Seems quite obvious that more objects means more memory usage.

Yes, but I mean when all those objects (namely, QSslSocket's) are destroyed, memory stays high. Even after few days it stays at 1.5 gb and doesn't go down.

Launch another program and have it allocate almost all of your physical memory and do something with all the memory (e.g. fill it with value '42'). Then see if amount of memory allocated to your server drops. It will because the system will reclaim memory from your program in order to give it to the other one.

Ok, tried that. It proved to be partially correct, as it looked like memory was deallocated with some delay. But still if I run a fresh instance of the server, I can run another program requiring 400 Mb of memory. If I use the server a little and close all client connections, wait and then run that other app, it will fail. So I don't know what I am missing. I was trying Visual Leak Detector, it reports 44 bytes leak:

Call Stack:
z:\testserver-ssl\main.cpp (17): testserver-ssl.exe!Server::Server + 0x7 bytes
z:\testserver-ssl\main.cpp (192): testserver-ssl.exe!main + 0x8 bytes
f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c (586): testserver-ssl.exe!__tmainCRTStartup + 0x19 bytes
f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c (403): testserver-ssl.exe!mainCRTStartup
0x7656919F (File and line number not available): KERNEL32.DLL!BaseThreadInitThunk + 0xE bytes
0x7703A22B (File and line number not available): ntdll.dll!RtlInitializeExceptionChain + 0x84 bytes
0x7703A201 (File and line number not available): ntdll.dll!RtlInitializeExceptionChain + 0x5A bytes

Strange, but not that much to care about.

Perhaps, you could suggest me some another leak detector as VLD seems to be Windows-only and I am developing the server that should run on Ubuntu Server.

[wysota]

On the test server there is 512 Mb.

Hmm... You want to handle 1k simultaneous clients with hardware that has 512MB of physical memory? That is going to be quite hard.

Perhaps, you could suggest me some another leak detector as VLD seems to be Windows-only and I am developing the server that should run on Ubuntu Server.

Valgrind

[mentalmushroom]

Hmm... You want to handle 1k simultaneous clients with hardware that has 512MB of physical memory? That is going to be quite hard.

Hmm... You want to handle 1k simultaneous clients with hardware that has 512MB of physical memory? That is going to be quite hard.

To copy to clipboard, switch view to plain text mode 

No-no, the server where the real application will run has 4 Gb RAM and we'll add more if needed. But there are two 'small' servers that I use for basic testing, researching etc. By the way, what amount of memory in your opinion the download server should have to handle several thousands of connections?

Valgrind

Valgrind

To copy to clipboard, switch view to plain text mode 

Ok, thanks. Although appears to be difficult to understand for me.

[wysota]

valgrind --tool=memcheck --leak-check=full yourappname yourappargs

[mentalmushroom]

Ok, this is what valgrind reports. I see some possible errors, but they all seem to point to Qt libs. Also the possible leak is ~14 Kb only, but I see the program takes much more. Could it be that QTcpServer, for example, takes much memory when there are many connections to it? As this object is not deleted when clients are disconnected.

==10703== HEAP SUMMARY:
==10703== in use at exit: 299,851 bytes in 4,365 blocks
==10703== total heap usage: 1,489,448 allocs, 1,485,083 frees, 4,536,782,939 bytes allocated
==10703==
==10703== 1,440 bytes in 6 blocks are possibly lost in loss record 896 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C6A8D: g_slist_prepend (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2A0: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F084D7: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4EFBB7D: QAbstractSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1E7F3: QSslSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x402339: Server::incomingConnection(int) (main.cpp:37)
==10703==
==10703== 1,488 bytes in 3 blocks are possibly lost in loss record 900 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AA245: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2B8: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F084D7: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4EFBB7D: QAbstractSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1E7F3: QSslSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x402339: Server::incomingConnection(int) (main.cpp:37)
==10703==
==10703== 1,680 bytes in 7 blocks are possibly lost in loss record 905 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C6A8D: g_slist_prepend (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2A0: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1EF88: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x530D280: QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703==
==10703== 1,920 bytes in 8 blocks are possibly lost in loss record 907 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C6A8D: g_slist_prepend (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2A0: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F085CA: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703==
==10703== 2,480 bytes in 5 blocks are possibly lost in loss record 912 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AA245: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2B8: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F085CA: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703==
==10703== 2,480 bytes in 5 blocks are possibly lost in loss record 913 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AA245: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2B8: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1EF88: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x530D280: QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703==
==10703== LEAK SUMMARY:
==10703== definitely lost: 0 bytes in 0 blocks
==10703== indirectly lost: 0 bytes in 0 blocks
==10703== possibly lost: 14,064 bytes in 45 blocks
==10703== still reachable: 285,787 bytes in 4,320 blocks
==10703== suppressed: 0 bytes in 0 blocks
==10703== Reachable blocks (those to which a pointer was found) are not shown.
==10703== To see them, rerun with: --leak-check=full --show-reachable=yes
==10703==
==10703== For counts of detected and suppressed errors, rerun with: -v
==10703== Use --track-origins=yes to see where uninitialised values come from
==10703== ERROR SUMMARY: 5389 errors from 14 contexts (suppressed: 2 from 2)

[yeye_olive]

IIRC some valgrind configurations exist for popular libraries like Qt that filter some of the noise caused by said libraries from valgrind's output.

You mentioned an increase in memory usage when clients start downloading, and you suspect that this memory is never freed. There could indeed be a leak in the code reading from/writing to sockets in your server, which we have yet to see.