Server programming - Best Practices

[Affenbrotbaum]

Hey guys,

I've been ordered to program a QT server that can handle multiple clients (~500 and up) at the same time. It's being used for bi-directional sending of strings to a subset of the connected clients (which depends on the type of message being sent).
I've looked around and came to the conclusion that a single-threaded asynchronous socket server is the way to go since resources are limited. I found an article in the wiki
( http://www.qtcentre.org/wiki/index.p...hout_threading ) that explains the basic setup of it. At the end,however, it states that such a server does not contain any security features.

Now my question is: What are the basic security features a public server (Did I mention that anybody will be able to register and login to the service the server will provide) should have ?

So far I can only come up with the obvious:
- only transferring critical user data in a hashed form
- Implement somewhat of a Ddos prevention that locks out certain IPs
- Clean up incoming strings right after reading them from a client and vice versa

Any other thoughts of things I have missed ? I have a feeling that I'm missing something big.

Any help is very much appreciated

Thank you
Alex

[tbscope]

The three points you mention above are just a very very small tip of the security iceberg. They are good starting points though.

Unfortunately, securing a server is extremely hard to do and requires constant updating.

Even if you have server software, listening to one or more ports, on a hardware machine, this doesn't prevent people from trying to access other ports on that machine. These ports might not be monitored by your software.
This means that you should look beyond your own software and incorporate the complete system.

Simply locking out an IP when you think a DDoS attack occurs will not always work. Some people are smart and make a DDoS attack look like a lot of normal requests. There are just so many that the server will not be able to handle them.

Other security threats are people who have access to your software and hardware. Are they able to set up user accounts? How can you protect this?

Important is software testing. Create a comprehensive set of tests to test everything you can think of and more before using the code for real. Bugs are the number one security vulnerability. Bugs can also exist in any hardware or other software you use. Choose them carefully.

And this list goes on ...

Check google for documentation on how to secure servers. I'm sure there exist some international standards, rules of thumb and books
It would also be a very good idea to get some education on the subject. Or to hire people who know how to deal with it, which in turn is in itself another security risk.

[Affenbrotbaum]

Hey tbscope,

Thanks for the hints. It doesn't have to be Armageddon-proof, just has to withstand the regular script-kiddy. Sourcecode and Hardware access are pretty limited as well since I'm the only guy running the show as of now. That's why the testing suite is a given as well. I don't wanna run the tests again and again....

Anyways, thanks again for all the help
Alex

[tbscope]

In that case, watch the input and output carefully. Create unittests.
Also learn about all the bugs in the libraries you use.

When you sell the software or use it in a production environment, be extremely carefull that you create a contract that states that you are not responsible for any bugs or errors or ...
If that's not in the contract, you will be very poor in no time. This is not something to handle lightly.

Edit: A collegue of my discovered a nice example of a bug today. Where I work, we have to sign in via Sharepoint everyday. This means that when I arrive, I have to fill in a form that I'm at my work place, and when I'm away I need to fill in where I will be going to etc... Now it seems that there is an rss feed available and I can just open that feed in a browser and see the detailed information of all my collegues. Not something that should be possible.

[wysota]

So far I can only come up with the obvious:
- only transferring critical user data in a hashed form
- Implement somewhat of a Ddos prevention that locks out certain IPs
- Clean up incoming strings right after reading them from a client and vice versa

"Security" is not only about intrusion prevention and hiding sensitive information. It's also about keeping your server responsive and functional. For instance preventing clients from keeping the connection open forever if they are not using the connection. Otherwise you'll run out of resources and it doesn't have to be caused by any malicious action (although of course it could be). You also have to be prepared to receive data and amount of data you don't expect. A simple example of a wrong approach would be something that's very often what people do in their servers:

void Server::onReadyRead() {
  if(!socket->canReadLine()) return;
  QByteArray data = socket->readLine();
  processRequest(data);
}

void Server::onReadyRead() {
  if(!socket->canReadLine()) return;
  QByteArray data = socket->readLine();
  processRequest(data);
}

To copy to clipboard, switch view to plain text mode 

What if you never get a newline? It can happen when the client doesn't send you a newline or if the request is larger than the socket buffer size and the newline is stuck on the sender's side.


There are many other similar things to consider.

[Affenbrotbaum]

Hey
@tbscope:
I am aware of the legal implications such an endeavor can have. Hence I made it very clear to my superiors that I need assurance that it won't backfire. Good point though, I should be putting that insurance in letters and get it signed.

@wysota:
Yes, that's why I didn't go for the multi-threaded server. It has way too much overhead. Not getting data in is also on the list since the solution will have a mobile component to it and those devices can go off-grid any time. I figured that something along the lines of

void Server::onReadyRead() {
    QTcpSocket *client = qobject_cast<QTcpSocket *>(sender());
    if(!client)
        return;
    int blockSize = 0 ;
    QString incomingData;
    QDataStream in(client);
    in.setVersion(QDataStream::Qt_4_7);
 
    if (blockSize == 0) {
        if (client->bytesAvailable() < (int)sizeof(quint16))
            return;
        in >> blockSize;
    }
    if (client->bytesAvailable() < blockSize)
        return;
    in >> incomingData;
    doStuff(incomingData);
}

void Server::onReadyRead() {
    QTcpSocket *client = qobject_cast<QTcpSocket *>(sender());
    if(!client)
        return;
    int blockSize = 0 ;
    QString incomingData;
    QDataStream in(client);
    in.setVersion(QDataStream::Qt_4_7);

    if (blockSize == 0) {
        if (client->bytesAvailable() < (int)sizeof(quint16))
            return;
        in >> blockSize;
    }
    if (client->bytesAvailable() < blockSize)
        return;
    in >> incomingData;
    doStuff(incomingData);
}

To copy to clipboard, switch view to plain text mode 

should do the job. On top of that is a timer that let's each connection time out after a set period of time. (The client pings the server to confirm its status)


Thanks for the advice, it's really helpful to get a different perspective on things.

Alex

[wysota]

@wysota:
Yes, that's why I didn't go for the multi-threaded server. It has way too much overhead.

Overhead is not relevant here. If you handle a connection one second later, it won't compromise anything. Not reacting to bogus or unexpected data will.

Not getting data in is also on the list since the solution will have a mobile component to it and those devices can go off-grid any time. I figured that something along the lines of

void Server::onReadyRead() {
    QTcpSocket *client = qobject_cast<QTcpSocket *>(sender());
    if(!client)
        return;
    int blockSize = 0 ;
    QString incomingData;
    QDataStream in(client);
    in.setVersion(QDataStream::Qt_4_7);
 
    if (blockSize == 0) {
        if (client->bytesAvailable() < (int)sizeof(quint16))
            return;
        in >> blockSize;
    }
    if (client->bytesAvailable() < blockSize)
        return;
    in >> incomingData;
    doStuff(incomingData);
}

void Server::onReadyRead() {
    QTcpSocket *client = qobject_cast<QTcpSocket *>(sender());
    if(!client)
        return;
    int blockSize = 0 ;
    QString incomingData;
    QDataStream in(client);
    in.setVersion(QDataStream::Qt_4_7);

    if (blockSize == 0) {
        if (client->bytesAvailable() < (int)sizeof(quint16))
            return;
        in >> blockSize;
    }
    if (client->bytesAvailable() < blockSize)
        return;
    in >> incomingData;
    doStuff(incomingData);
}

To copy to clipboard, switch view to plain text mode 

should do the job. On top of that is a timer that let's each connection time out after a set period of time. (The client pings the server to confirm its status)


Thanks for the advice, it's really helpful to get a different perspective on things.

If you create a QDataStream in a slot connected to readyRead() then I can immediately tell you your code is incorrect without even reading it.

[Affenbrotbaum]

Excuse my ignorance wysota, but what's wrong with it? I've done testing on it and seems to work with multiple concurrent connections without leaking any memory.

[tbscope]

A QDataStream must be read in its entirety.
You create a new data stream for each chunk which will never work.

[wysota]

Excuse my ignorance wysota, but what's wrong with it? I've done testing on it and seems to work with multiple concurrent connections without leaking any memory.

It works only when you receive data you expect in a form you expect and amount you expect.

For example you are reading a block size and then discarding all data already read if there is not enough data available in the socket. Next time you get the signal when the rest of the data arrives but since you are not waiting for the data anymore, you will try to synchronize the data stream again which will obviously fail because there is some data from the previous block still left in the socket.

And provided you even fix it and have a buffer that will store all the incoming data before processing it, what if someone sends you a block size of 2GB? Are you going to buffer 2GB of data in your application for each incoming connection? What if you get 10 such connections? Do you have 20GB of memory in your machine?

If you have to use QDataStream, you synchronize it only once when the connection is established and you should closely monitor the status of the stream and at the first sign of lost synchro, you have to immediately drop the connection.

[Affenbrotbaum]

Hey

Alright I fixed it now and only one instance of the Datastream exists now per open Socket.
We are mainly dealing with small data transfer (< 5kb per hour per client). It automatically refuses all block sizes that are above a certain threshold. If the incoming data is incomplete waitForReadyRead() will take care of it. I'm doing some reworking right now, but I can post the code later.

Again, thanks for all the help. It's much appreciated
Alex

[wysota]

If the incoming data is incomplete waitForReadyRead() will take care of it.

That's a really bad idea. You are blocking your whole application while waiting (maybe indefinitely) for data from one client.

Again, don't focus on what you expect. Instead expect the unexpected.

[Affenbrotbaum]

Yeah I noticed that during testing and now I'm a little stomped. I guess there is no asynchronous implementation of that function ?

[wysota]

Sure there is. It's the readyRead() signal

[Affenbrotbaum]

Hehe ok now I'm confused . The above code in post #6 is what's connected to the readyRead() signal. Only 2 differences are a) The QDataStream now gets created once per socket when I'm working through the TcpServer::hasPendingConnections() list and b) I limit the data being received since I know all the strings and their length that are permitted to be submitted (socket->bytesAvailable()). I just had for testing purposes a waitForReadyRead() in there to see how it behaves with large or no data coming in.

So far, everything seems to be working now. I'll post some code this weekend so other people have an example of this whole conversation. Thank you very much for all the help

Cheers
Alex

[wysota]

The above code in post #6 is what's connected to the readyRead() signal.

That's correct and that's the right approach. But the code you put inside the slot connected to this signal is relevant.

b) I limit the data being received since I know all the strings and their length that are permitted to be submitted (socket->bytesAvailable()).

And that's your problem. You assume the data is going to look like you expect it. A good server is not one that behaves correctly for the data it expects, that's easy to achieve. A good server behaves correctly for the data (and behaviour) it doesn't expect.

I just had for testing purposes a waitForReadyRead() in there to see how it behaves with large or no data coming in.

So far, everything seems to be working now.

Such tests are useless.

With the code you posted so far I can compromise your whole server computer (not only your application) within seconds.