QTcpSocket/QTcpServer on a heavy load server

[Eumcoz]

Hello, I am designing and making a server that should be able to handle about 100+ hits per second. The information I am getting from the server is just the HTTP header. Based on the information from the header, it will query a database(different thread) for some information and send the final information back to the QTcpServer which create an output string, and send back a HTTP Response. I am having a big problem with this that I cannot debug. My code look similar to this:

TCPInterface::TCPInterface(QObject *parent): QTcpServer(parent)
{
    //start listening for tcp traffic on port 80
    listen(QHostAddress::Any, 80);
 
    connect(this,SIGNAL(sendInfo(QTcpSocket*, QString *)), databaseThread, SLOT(recieveInfo(QTcpSocket*, QString*)));
    connect(databaseThread, SIGNAL(sendToTCPSend(QTcpSocket *, QString *)), this, SLOT(TCPSend(QTcpSocket*, QString*)));
 
}

TCPInterface::TCPInterface(QObject *parent): QTcpServer(parent)
{
    //start listening for tcp traffic on port 80
    listen(QHostAddress::Any, 80);

    connect(this,SIGNAL(sendInfo(QTcpSocket*, QString *)), databaseThread, SLOT(recieveInfo(QTcpSocket*, QString*)));
    connect(databaseThread, SIGNAL(sendToTCPSend(QTcpSocket *, QString *)), this, SLOT(TCPSend(QTcpSocket*, QString*)));

}

To copy to clipboard, switch view to plain text mode 

void TCPInterface::incomingConnection(int socket)
{
    QTcpSocket *s = new QTcpSocket(this);
 
    connect(s, SIGNAL(readyRead()), this, SLOT(readClient()));
//    connect(s, SIGNAL(disconnected()), this, SLOT(discardClient()));
 
    s->setSocketDescriptor(socket);
}

void TCPInterface::incomingConnection(int socket)
{
    QTcpSocket *s = new QTcpSocket(this);

    connect(s, SIGNAL(readyRead()), this, SLOT(readClient()));
//    connect(s, SIGNAL(disconnected()), this, SLOT(discardClient()));

    s->setSocketDescriptor(socket);
}

To copy to clipboard, switch view to plain text mode 

//void TCPInterface::discardClient()
//{
//    QTcpSocket* socket = (QTcpSocket*)sender();
//    socket->deleteLater();
//}

//void TCPInterface::discardClient()
//{
//    QTcpSocket* socket = (QTcpSocket*)sender();
//    socket->deleteLater();
//}

To copy to clipboard, switch view to plain text mode 

void TCPInterface::readClient()
{
    QTcpSocket* socket = (QTcpSocket*)sender();
 
    QString header;
    while(socket->canReadLine())
    {
        header += socket->readLine();
    }
 
    emit sendInfo(socket, headerInfo);
}

void TCPInterface::readClient()
{
    QTcpSocket* socket = (QTcpSocket*)sender();

    QString header;
    while(socket->canReadLine())
    {
        header += socket->readLine();
    }

    emit sendInfo(socket, headerInfo);
}

To copy to clipboard, switch view to plain text mode 

void databaseThread::recieveInfo(QTcpSocket* socket, QString* headerInfo)
{
   QString*outputInfo = getDatabaseInfo(headerInfo);
   emit sendToTCPSend(socket, outputInfo);
}

void databaseThread::recieveInfo(QTcpSocket* socket, QString* headerInfo)
{
   QString*outputInfo = getDatabaseInfo(headerInfo);
   emit sendToTCPSend(socket, outputInfo);
}

To copy to clipboard, switch view to plain text mode 

void TCPInterface::TCPSend(QTcpSocket* socket, QString* outputInfo);
{
    QString response = "HTTP/1.0 200 Ok\r\n";
    response += "Content-Type: text/html; charset=\"utf-8\"\r\n";
    response += "\r\n" + *outputInfo + "\n";
 
    if(socket->isWritable() && socket->isOpen())
    {
         socket->write(response.toAscii());
    }
    //socket->disconnectFromHost();
    socket->close();
    delete headerInfo;
}

void TCPInterface::TCPSend(QTcpSocket* socket, QString* outputInfo);
{
    QString response = "HTTP/1.0 200 Ok\r\n";
    response += "Content-Type: text/html; charset=\"utf-8\"\r\n";
    response += "\r\n" + *outputInfo + "\n";
    
    if(socket->isWritable() && socket->isOpen())
    {
         socket->write(response.toAscii());
    }
    //socket->disconnectFromHost();
    socket->close();
    delete headerInfo;
}

To copy to clipboard, switch view to plain text mode 

I having one main problem which I have an idea what it is, but cannot find a solution to fix it.

My problem is my memory is constantly increasing as I get more hits. I am sure the cause of this is my QTcpSockets are never being deleted, since I am just closing them. However when I don't use close, and use disconnectFromHost and disconnected/discardClient slot/signal my server will crash with heavy traffic(no message or anything so I am not sure of the exact reason of the crash). Has anyone run into this problem before? Any ideas at all.

[StrikeByte]

I think the problem is that you try to delete the socket when it is not ready sending/receiving data
connect to the signal disconnected() of the QTcpSocket use disconnectFromHost() to initiate the disconnect.
then when the signal is fired disconnect the signal and kill the QTcpSocket

[spirit]

Side note to StrikeByte's. Kill them using QObject::deleteLater.

[Eumcoz]

Sorry, I think I made everything kind of confusing. I have tried using deleteLater.

I have tried:

void TCPInterface::discardClient()
{
   QTcpSocket* socket = (QTcpSocket*)sender();
   socket->deleteLater();
}

void TCPInterface::discardClient()
{
   QTcpSocket* socket = (QTcpSocket*)sender();
   socket->deleteLater();
}

To copy to clipboard, switch view to plain text mode 

Which is connected to the signal disconnected. And when doing this I call disconnectFromHost() instead of close() in my last function.

When using this method, my program will crash. So i am assuming something weird is happening when I delete the socket, but I have no clue as to what it is.

Also another note. It does not crash every time on delete. I am stress testing my server with a script that is constantly sending me get requests, and It will crash anywhere between the 2nd and 200th packet. When I see a lower amount of traffic going into the server, I have no problem with the server staying up. Any ideas what the cause of this problem could be?

[spirit]

As I understand, you call "deleteLater" then "disconnectFromHost"? You should try to call "disconnectFromHost" and then "deleteLater".

[Eumcoz]

No, I call disconnectFromHost first. deleteLater is in my slot connected to disconnected signal.

But that did get me thinking. What if a client disconnected from me while I was doing work(database / response message). I should get a disconnected signal, which will delete the socket(via deleteLater). If I then try and do:

if(socket->isWritable() && socket->isOpen())

if(socket->isWritable() && socket->isOpen())

To copy to clipboard, switch view to plain text mode 

socket would be an invalid pointer, and then crash my program. Is there anyway to protect myself from this?

---

Added after 22 minutes:

Well, that was defiantly a problem. I set a slot to handle the destroyed() signal of the socket, and set the pointer to NULL there. Then I checked to see if the pointer is NULL before I use it. That did not fix my problem unfortunately, my program will still crash under heavy load.

[StrikeByte]

Try using qobject_cast<QtcpSocket*>(sender()); it will return null if the sender can't be converted (maybe the sender isn't the tcpsocket)

[Eumcoz]

Try using qobject_cast<QtcpSocket*>(sender()); it will return null if the sender can't be converted (maybe the sender isn't the tcpsocket)

That fixed it, not sure why the sender wouldn't have been a tcpsocket, but it worked. Thank you very much!

[Eumcoz]

Actually, I had a typo that prevented the delete. I simplified the problem as much as I think I could. My new train of thought is that my server crash has something to do with the Event Loop.

I have attached a simple program that replicates the problem, however you need to have a client that will disconnect from the server before the response comes back or I don't think the problem will happen. If you don't want to download the program here's the guts of the program.

TCPInterface::TCPInterface(QObject *parent) : QTcpServer(parent)
{
    listen(QHostAddress::Any,80);
    connect(this, SIGNAL(toDB(QTcpSocket*)), App::getInstance()->getDatabase(), SLOT(fromInterface(QTcpSocket*)), Qt::QueuedConnection);
    connect(App::getInstance()->getDatabase(), SIGNAL(sendToInterface(QTcpSocket*)), this, SLOT(fromDB(QTcpSocket*)), Qt::QueuedConnection);
}
 
void TCPInterface::incomingConnection(int socket)
{
    QTcpSocket *s = new QTcpSocket(this);
    connect(s, SIGNAL(readyRead()), this, SLOT(readClient()));
    connect(s, SIGNAL(disconnected()), this, SLOT(discardClient()));
    s->setSocketDescriptor(socket);
}
 
void TCPInterface::readClient()
{
    QTcpSocket* socket = (QTcpSocket*)sender();
 
    emit toDB(socket);
}
 
void TCPInterface::discardClient()
{
    QTcpSocket* socket = (QTcpSocket*)sender();
    socket->deleteLater();
}
 
void TCPInterface::fromDB(QTcpSocket *socket)
{
    QString tmp = "HTTP/1.0 200 Ok\r\n";
    tmp += "Content-Type: text/html; charset=\"utf-8\"\r\n";
    tmp += "\r\nTest\n";
 
 
    socket->write(tmp.toAscii());
    socket->disconnectFromHost();
}

TCPInterface::TCPInterface(QObject *parent) : QTcpServer(parent)
{
    listen(QHostAddress::Any,80);
    connect(this, SIGNAL(toDB(QTcpSocket*)), App::getInstance()->getDatabase(), SLOT(fromInterface(QTcpSocket*)), Qt::QueuedConnection);
    connect(App::getInstance()->getDatabase(), SIGNAL(sendToInterface(QTcpSocket*)), this, SLOT(fromDB(QTcpSocket*)), Qt::QueuedConnection);
}

void TCPInterface::incomingConnection(int socket)
{
    QTcpSocket *s = new QTcpSocket(this);
    connect(s, SIGNAL(readyRead()), this, SLOT(readClient()));
    connect(s, SIGNAL(disconnected()), this, SLOT(discardClient()));
    s->setSocketDescriptor(socket);
}

void TCPInterface::readClient()
{
    QTcpSocket* socket = (QTcpSocket*)sender();

    emit toDB(socket);
}

void TCPInterface::discardClient()
{
    QTcpSocket* socket = (QTcpSocket*)sender();
    socket->deleteLater();
}

void TCPInterface::fromDB(QTcpSocket *socket)
{
    QString tmp = "HTTP/1.0 200 Ok\r\n";
    tmp += "Content-Type: text/html; charset=\"utf-8\"\r\n";
    tmp += "\r\nTest\n";


    socket->write(tmp.toAscii());
    socket->disconnectFromHost();
}

To copy to clipboard, switch view to plain text mode 

and

void DatabaseInterface::fromInterface(QTcpSocket *socket)
{
    emit sendToInterface(socket);
}

void DatabaseInterface::fromInterface(QTcpSocket *socket)
{
    emit sendToInterface(socket);
}

To copy to clipboard, switch view to plain text mode 

Some things I have noted
1) The script I am using to test my server will often disconnect before a response is sent.
2) The server will not crash unless the socket is passed to a different thread.
3) From the backtrace it seems like the crash has something to do with the event loop.

I am assuming that since my QTcpSocket my server is crashing on the client disconnecting because the of the different event loops, but not sure of the exact reason. Has anyone had this problem before? Is there a better way I should be designing my server to avoid this problem?

Here is the backtrace that I got when my server crashed.

Error: signal 11:
./SimpleTCPServer[0x401ce6]
/lib64/libc.so.6[0x3c14032920]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN9QIODevice5writeEPKcx+0x22)[0x7fac6b5a6d22]
./SimpleTCPServer[0x401fb6]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN7QObject5eventEP6QEvent+0x38e)[0x7fac6b62cbae]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN16QCoreApplication14notifyIntern alEP7QObjectP6QEvent+0x8c)[0x7fac6b61a30c]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN23QCoreApplicationPrivate16sendP ostedEventsEP7QObjectiP11QThreadData+0x3d3)[0x7fac6b61ebf3]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(+0x1c7cf3)[0x7fac6b64bcf3]
/lib64/libglib-2.0.so.0(g_main_context_dispatch+0x22e)[0x3c15438f0e]
/lib64/libglib-2.0.so.0[0x3c1543c938]
/lib64/libglib-2.0.so.0(g_main_context_iteration+0x7a)[0x3c1543ca3a]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN20QEventDispatcherGlib13processE ventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0 x73)[0x7fac6b64b833]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN10QEventLoop13processEventsE6QFl agsINS_17ProcessEventsFlagEE+0x32)[0x7fac6b618ec2]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN10QEventLoop4execE6QFlagsINS_17P rocessEventsFlagEE+0x164)[0x7fac6b619334]
/home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN16QCoreApplication4execEv+0xb9)[0x7fac6b61efc9]
./SimpleTCPServer[0x401c95]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3c1401ecdd]
./SimpleTCPServer[0x401b59]

[spirit]

Can you also add your script to test the app?

[Eumcoz]

Sure I am using a simple php script:

<?php
//$host = '10.1.10.36';
$host = '10.1.10.43';
 
$port = 80;
 
$service_uri = '/7';
 
while (true) {
    $fp = fsockopen($host, $port) or die ("Could not open a connection to host <i>$host</i> on port <i>$port</i>.");
 
    # compose HTTP request header
    $header = "Watch Your Mouth";
    fputs($fp, "GET $service_uri  HTTP/1.1\r\n");
	fputs($fp, "GET \r\n");
 
    fputs($fp, $header);
 
}
 
 
?>

<?php
//$host = '10.1.10.36';
$host = '10.1.10.43';

$port = 80;

$service_uri = '/7';

while (true) {
    $fp = fsockopen($host, $port) or die ("Could not open a connection to host <i>$host</i> on port <i>$port</i>.");

    # compose HTTP request header
    $header = "Watch Your Mouth";
    fputs($fp, "GET $service_uri  HTTP/1.1\r\n");
	fputs($fp, "GET \r\n");
        
    fputs($fp, $header);

}


?>

To copy to clipboard, switch view to plain text mode 

Attached is also a slightly better version of the server that makes sure that a socket is valid(has not been destroyed), before using the socket.

And also, the server does not crash right away, the more traffic going threw it, the more of a chance it will crash.