QSslSocket memory leak

[mentalmushroom]

Hello. I am developing the download server that should handle 1000+ simultaneous connections. I noticed some memory issue when there are many connections (well, perhaps, it happens with a small number of connections either, but it's just not that obvious). So I see not all the memory that was allocated while establishing connections is released when they are closed.

Here goes the simple server code. It doesn't send or receive any data.

#include <QCoreApplication>
#include <QtNetwork>
 
class Server: public QTcpServer
{
    Q_OBJECT
public:
    Server()
    {
        this->total = 0;
    }
 
protected:
    virtual void incomingConnection(int d);
 
private:
    int total;
 
private slots:
    void handleClientDisconnected();
};
 
void Server::incomingConnection(int d)
{
    QSslSocket *sock = new QSslSocket;
    if (!sock->setSocketDescriptor(d))    
        delete sock;
 
    qDebug() << "socket " << d << " connected, total: " << ++this->total;
 
    Q_ASSERT(QFile::exists(":/ssl/resources/my.crt"));
    sock->setLocalCertificate(":/ssl/resources/my.crt");
    sock->setPrivateKey(":/ssl/resources/my.key", QSsl::Rsa, QSsl::Pem, "la-la-la");
 
    connect(sock, SIGNAL(disconnected()), this, SLOT(handleClientDisconnected()));
 
    sock->startServerEncryption();
}
 
void Server::handleClientDisconnected()
{    
    QSslSocket *sock = static_cast<QSslSocket*>(sender());
    qDebug() << "Client " << sock->socketDescriptor() << " disconnected, total: " << --this->total;
    sock->deleteLater(); // ! deleting the disconnected socket  
}
 
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
 
    Server server;
    if (server.listen(QHostAddress::Any, 563))
        qDebug() << "listen started";
    else qDebug() << "listen failed";
 
    return a.exec();
}
 
#include "main.moc"

#include <QCoreApplication>
#include <QtNetwork>

class Server: public QTcpServer
{
    Q_OBJECT
public:
    Server()
    {
        this->total = 0;
    }

protected:
    virtual void incomingConnection(int d);

private:
    int total;

private slots:
    void handleClientDisconnected();
};

void Server::incomingConnection(int d)
{
    QSslSocket *sock = new QSslSocket;
    if (!sock->setSocketDescriptor(d))    
        delete sock;
    
    qDebug() << "socket " << d << " connected, total: " << ++this->total;

    Q_ASSERT(QFile::exists(":/ssl/resources/my.crt"));
    sock->setLocalCertificate(":/ssl/resources/my.crt");
    sock->setPrivateKey(":/ssl/resources/my.key", QSsl::Rsa, QSsl::Pem, "la-la-la");

    connect(sock, SIGNAL(disconnected()), this, SLOT(handleClientDisconnected()));

    sock->startServerEncryption();
}

void Server::handleClientDisconnected()
{    
    QSslSocket *sock = static_cast<QSslSocket*>(sender());
    qDebug() << "Client " << sock->socketDescriptor() << " disconnected, total: " << --this->total;
    sock->deleteLater(); // ! deleting the disconnected socket  
}

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    Server server;
    if (server.listen(QHostAddress::Any, 563))
        qDebug() << "listen started";
    else qDebug() << "listen failed";

    return a.exec();
}

#include "main.moc"

To copy to clipboard, switch view to plain text mode 

This is how client connections are opened:

QSslSocket* sock[1200];
 
for (int i = 0; i < 500; ++i)
{
 sock[i] = new QSslSocket; 
 sock[i]->connectToHostEncrypted("mytestserver.com", 563);
}

QSslSocket* sock[1200];

for (int i = 0; i < 500; ++i)
{
 sock[i] = new QSslSocket; 
 sock[i]->connectToHostEncrypted("mytestserver.com", 563);
}

To copy to clipboard, switch view to plain text mode 

Again, now data is sent or received. I was running the test several times establishing and closing 200-500 connections. Now there is around 60 Mb of memory used by the server, although all connections were closed.

What is wrong in my server code, so it leaks much? The leak is bigger when transferring some data.

[yeye_olive]

Firstly, how do you know your server leaks memory? Just looking at the process' memory usage is a poor indicator, as the C++ runtime manages its own allocation. There is a huge difference between a delete and the deallocation of memory pages. Have you tried running your server through valgrind?

Secondly, your current code looks fragile to me. You only delete a socket upon receving its disconnected() signal. Are you certain you always receive it? What happens if you destroy the Server object first? Most likely, the connection from the sockets' disconnected() signal to the server's handleClientDisconnected() will be removed, and your sockets will leak. You could make the Server object the parent of the socket with QSslSocket *sock = new QSslSocket(this); or use QTcpServer::nextPendingConnection() which does that for you.

By the way, does your client delete its 500 sockets?

As a general rule, you should prevent resources from leaking by registering them with a manager. For QObject and derived classes, you can provide a parent upon creation. For other heap-allocated objects, the standard C++ library has a few useful smart pointer classes. This should be a real discipline, especially when exceptions and callbacks into user code lie around. Look up the RAII design pattern for more details.

[mentalmushroom]

Firstly, how do you know your server leaks memory? Just looking at the process' memory usage is a poor indicator, as the C++ runtime manages its own allocation.

I use top command to check the process memory. And I see it has grown to 60 Mb after closing all the connections.

There is a huge difference between a delete and the deallocation of memory pages.

Yes, I was reading about that, but I need somehow to deallocate that memory as it grows much when many clients are downloading.

Secondly, your current code looks fragile to me. You only delete a socket upon receving its disconnected() signal. Are you certain you always receive it? What happens if you destroy the Server object first? Most likely, the connection from the sockets' disconnected() signal to the server's handleClientDisconnected() will be removed, and your sockets will leak.

Well, this is a test sample, so I don't really care of what happens when the server object is destroyed first. I see memory growing without destroying it. And yes, I am sure I receive disconnected() signal as I was checking it with the counter, so when it turns to zero memory is not completely returned.

By the way, does your client delete its 500 sockets?

Does it matter? I am developing the server, not the client. And I want to fix the server issue regardless of the way the client works.

As a general rule, you should prevent resources from leaking by registering them with a manager. For QObject and derived classes, you can provide a parent upon creation. For other heap-allocated objects, the standard C++ library has a few useful smart pointer classes. This should be a real discipline, especially when exceptions and callbacks into user code lie around. Look up the RAII design pattern for more details.

I will take it into account, but it seems to me every socket gets deleted, but the memory is not always or not completely returned after that.

[wysota]

Yes, I was reading about that, but I need somehow to deallocate that memory as it grows much when many clients are downloading.

How much memory do you have that you are worried your kernel wants to keep those 60 megabytes allocated for you in case you will want them back in a couple of seconds? Did you see what happens when your system runs out of free memory? That is when it should take the memory away from you and not when you want it to. It knows better how expensive it is to allocate or deallocate memory to a process.

If you claim your application leaks memory then use tools for checking memory leaks and not tools reporting memory mapping.

[mentalmushroom]

How much memory do you have that you are worried your kernel wants to keep those 60 megabytes allocated for you in case you will want them back in a couple of seconds?

Well, 60 Mb this is what I got after few test runs. If I continue it will grow more. It will grow much more when there are 1000+ clients downloading. By the way, when there was much data sent to the socket it looks like the memory used for buffering is also not returned.

Did you see what happens when your system runs out of free memory?

Honestly, no, but in some cases it reached 1.5 gb (reported by top) and it doesn't really drop even when all clients are disconnected. However, I noticed memory may not grow, e.g. when I try to establish less connections then there were established before. So could it be the memory is not deallocated on purpose just to save future allocation time or something like that?

[wysota]

Well, 60 Mb this is what I got after few test runs.

I asked how much RAM you had in your computer, not how much you saw allocated to your process.

If I continue it will grow more. It will grow much more when there are 1000+ clients downloading.

Seems quite obvious that more objects means more memory usage.

By the way, when there was much data sent to the socket it looks like the memory used for buffering is also not returned.

You seem to miss the point. The fact that you call "free" or "delete" on your data doesn't mean the system will deallocate memory from your process. If you use much memory in a short period of time the system will notice that and will keep the memory allocated to your process in case yo want to use more memory again.

Honestly, no, but in some cases it reached 1.5 gb (reported by top) and it doesn't really drop even when all clients are disconnected.

Launch another program and have it allocate almost all of your physical memory and do something with all the memory (e.g. fill it with value '42'). Then see if amount of memory allocated to your server drops. It will because the system will reclaim memory from your program in order to give it to the other one.

So could it be the memory is not deallocated on purpose just to save future allocation time or something like that?

Congratulations, now read our posts again.

[mentalmushroom]

I asked how much RAM you had in your computer, not how much you saw allocated to your process.

On the test server there is 512 Mb. I thought you mean why am I so worried about 60 Mb. I just tried to say memory grows significantly if I test it well, so this is why I care about it.

Seems quite obvious that more objects means more memory usage.

Yes, but I mean when all those objects (namely, QSslSocket's) are destroyed, memory stays high. Even after few days it stays at 1.5 gb and doesn't go down.

Launch another program and have it allocate almost all of your physical memory and do something with all the memory (e.g. fill it with value '42'). Then see if amount of memory allocated to your server drops. It will because the system will reclaim memory from your program in order to give it to the other one.

Ok, tried that. It proved to be partially correct, as it looked like memory was deallocated with some delay. But still if I run a fresh instance of the server, I can run another program requiring 400 Mb of memory. If I use the server a little and close all client connections, wait and then run that other app, it will fail. So I don't know what I am missing. I was trying Visual Leak Detector, it reports 44 bytes leak:

Call Stack:
z:\testserver-ssl\main.cpp (17): testserver-ssl.exe!Server::Server + 0x7 bytes
z:\testserver-ssl\main.cpp (192): testserver-ssl.exe!main + 0x8 bytes
f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c (586): testserver-ssl.exe!__tmainCRTStartup + 0x19 bytes
f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c (403): testserver-ssl.exe!mainCRTStartup
0x7656919F (File and line number not available): KERNEL32.DLL!BaseThreadInitThunk + 0xE bytes
0x7703A22B (File and line number not available): ntdll.dll!RtlInitializeExceptionChain + 0x84 bytes
0x7703A201 (File and line number not available): ntdll.dll!RtlInitializeExceptionChain + 0x5A bytes

Strange, but not that much to care about.

Perhaps, you could suggest me some another leak detector as VLD seems to be Windows-only and I am developing the server that should run on Ubuntu Server.

[wysota]

On the test server there is 512 Mb.

Hmm... You want to handle 1k simultaneous clients with hardware that has 512MB of physical memory? That is going to be quite hard.

Perhaps, you could suggest me some another leak detector as VLD seems to be Windows-only and I am developing the server that should run on Ubuntu Server.

Valgrind

[mentalmushroom]

Hmm... You want to handle 1k simultaneous clients with hardware that has 512MB of physical memory? That is going to be quite hard.

Hmm... You want to handle 1k simultaneous clients with hardware that has 512MB of physical memory? That is going to be quite hard.

To copy to clipboard, switch view to plain text mode 

No-no, the server where the real application will run has 4 Gb RAM and we'll add more if needed. But there are two 'small' servers that I use for basic testing, researching etc. By the way, what amount of memory in your opinion the download server should have to handle several thousands of connections?

Valgrind

Valgrind

To copy to clipboard, switch view to plain text mode 

Ok, thanks. Although appears to be difficult to understand for me.

[wysota]

valgrind --tool=memcheck --leak-check=full yourappname yourappargs

[mentalmushroom]

Ok, this is what valgrind reports. I see some possible errors, but they all seem to point to Qt libs. Also the possible leak is ~14 Kb only, but I see the program takes much more. Could it be that QTcpServer, for example, takes much memory when there are many connections to it? As this object is not deleted when clients are disconnected.

==10703== HEAP SUMMARY:
==10703== in use at exit: 299,851 bytes in 4,365 blocks
==10703== total heap usage: 1,489,448 allocs, 1,485,083 frees, 4,536,782,939 bytes allocated
==10703==
==10703== 1,440 bytes in 6 blocks are possibly lost in loss record 896 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C6A8D: g_slist_prepend (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2A0: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F084D7: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4EFBB7D: QAbstractSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1E7F3: QSslSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x402339: Server::incomingConnection(int) (main.cpp:37)
==10703==
==10703== 1,488 bytes in 3 blocks are possibly lost in loss record 900 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AA245: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2B8: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F084D7: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4EFBB7D: QAbstractSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1E7F3: QSslSocket::setSocketDescriptor(int, QAbstractSocket::SocketState, QFlags<QIODevice::OpenModeFlag>) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x402339: Server::incomingConnection(int) (main.cpp:37)
==10703==
==10703== 1,680 bytes in 7 blocks are possibly lost in loss record 905 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C6A8D: g_slist_prepend (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2A0: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1EF88: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x530D280: QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703==
==10703== 1,920 bytes in 8 blocks are possibly lost in loss record 907 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C6A8D: g_slist_prepend (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2A0: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F085CA: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703==
==10703== 2,480 bytes in 5 blocks are possibly lost in loss record 912 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AA245: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2B8: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x5316189: QSocketNotifier::QSocketNotifier(int, QSocketNotifier::Type, QObject*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F085CA: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703==
==10703== 2,480 bytes in 5 blocks are possibly lost in loss record 913 of 940
==10703== at 0x4C29BE8: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x4C29C97: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==10703== by 0x677F0C9: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67C60C3: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AA245: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x67AB2B8: g_source_add_poll (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4)
==10703== by 0x53287A9: QEventDispatcherGlib::registerSocketNotifier(QSock etNotifier*) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F0093B: QAbstractSocket::writeData(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x528BCAE: QIODevice::write(char const*, long long) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703== by 0x4F24C6E: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x4F1EF88: ??? (in /usr/lib/x86_64-linux-gnu/libQtNetwork.so.4.8.1)
==10703== by 0x530D280: QMetaObject::activate(QObject*, QMetaObject const*, int, void**) (in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1)
==10703==
==10703== LEAK SUMMARY:
==10703== definitely lost: 0 bytes in 0 blocks
==10703== indirectly lost: 0 bytes in 0 blocks
==10703== possibly lost: 14,064 bytes in 45 blocks
==10703== still reachable: 285,787 bytes in 4,320 blocks
==10703== suppressed: 0 bytes in 0 blocks
==10703== Reachable blocks (those to which a pointer was found) are not shown.
==10703== To see them, rerun with: --leak-check=full --show-reachable=yes
==10703==
==10703== For counts of detected and suppressed errors, rerun with: -v
==10703== Use --track-origins=yes to see where uninitialised values come from
==10703== ERROR SUMMARY: 5389 errors from 14 contexts (suppressed: 2 from 2)