QDataStream loses data over QTcpSocket?

[KShots]

This has been bugging me for a while... I'm attempting to send a QImage over a QDataStream (initialized over a QTcpSocket), and finding that it's a lot more difficult than I thought.

I've verified that the QDataStream is functional (it is properly sending XML content over the wire and is received intact), and I verified that the image is not null when it is sent... but when I attempt to send over a QImage, I get nothing, or at worst, partial data. I've tried something as simple as this for the sender:

QDataStream ds(&sock);
ds << myImage;

QDataStream ds(&sock);
ds << myImage;

To copy to clipboard, switch view to plain text mode 

... with the receiver doing something as simple as this:

QDataStream ds(&sock);
ds >> myImage;

QDataStream ds(&sock);
ds >> myImage;

To copy to clipboard, switch view to plain text mode 

... which results in a null image.

I then tried to get fancy and use the rawData() functions to send the beast over:

char *
readRawData(char *buffer, QDataStream &ds, int length)
{
	int bytesRead, bytesTotal(0);
	for(; bytesTotal < length; bytesTotal += bytesRead)
	{
		if((bytesRead = ds.readRawData(buffer + bytesTotal, length - bytesTotal)) == -1)
		{
			qFatal("%s:%s(%d) - failed to read raw data: %s", __FILE__, __FUNCTION__, __LINE__, ds.device()->errorString().toLocal8Bit().constData());
		}
		if(!bytesRead)
		{
			qFatal("%s:%s(%d) - broken connection while reading raw data", __FILE__, __FUNCTION__, __LINE__);
		}
	}
	return buffer;
}
 
const char *
writeRawData(const char *buffer, QDataStream &ds, int size)
{
	int bytesWritten, bytesTotal(0);
	for(; bytesTotal < size; bytesTotal += bytesWritten)
	{
		if((bytesWritten = ds.writeRawData(buffer + bytesTotal, size - bytesTotal)) == -1)
		{
			qFatal("%s:%s(%d) - Failed to write raw bytes: %s", __FILE__, __FUNCTION__, __LINE__, ds.device()->errorString().toLocal8Bit().constData());
		}
		if(!bytesWritten)
		{
			qFatal("%s:%s(%d) - stream closed while writing raw data", __FILE__, __FUNCTION__, __LINE__);
		}
	}
	return buffer;
}

char *
readRawData(char *buffer, QDataStream &ds, int length)
{
	int bytesRead, bytesTotal(0);
	for(; bytesTotal < length; bytesTotal += bytesRead)
	{
		if((bytesRead = ds.readRawData(buffer + bytesTotal, length - bytesTotal)) == -1)
		{
			qFatal("%s:%s(%d) - failed to read raw data: %s", __FILE__, __FUNCTION__, __LINE__, ds.device()->errorString().toLocal8Bit().constData());
		}
		if(!bytesRead)
		{
			qFatal("%s:%s(%d) - broken connection while reading raw data", __FILE__, __FUNCTION__, __LINE__);
		}
	}
	return buffer;
}

const char *
writeRawData(const char *buffer, QDataStream &ds, int size)
{
	int bytesWritten, bytesTotal(0);
	for(; bytesTotal < size; bytesTotal += bytesWritten)
	{
		if((bytesWritten = ds.writeRawData(buffer + bytesTotal, size - bytesTotal)) == -1)
		{
			qFatal("%s:%s(%d) - Failed to write raw bytes: %s", __FILE__, __FUNCTION__, __LINE__, ds.device()->errorString().toLocal8Bit().constData());
		}
		if(!bytesWritten)
		{
			qFatal("%s:%s(%d) - stream closed while writing raw data", __FILE__, __FUNCTION__, __LINE__);
		}
	}
	return buffer;
}

To copy to clipboard, switch view to plain text mode 

... which, when viewed in a debugger, shows that I get partial data - the size of the image is 1536000 bytes, and I get 196433 bytes when I do this (then it claims the stream was closed because it can't read anything further). What gives?

[wysota]

What gives?

Why did you expect to receive all data at once?

[squidge]

You need to learn how TCP works - http://en.wikipedia.org/wiki/Transmi...ntrol_Protocol

Then you will understand that you can't simply expect to send a chunk of data and receive it at the other side in one go.

Preferably, you should packetise the data so you know more about whats happening during the transfer.

[KShots]

Yes, I'm aware that I can't send it all in one shot - that's why I wrote the read and write functions like I did, they would keep writing until it's all over the wire, and keep reading until it has all the data it expects. I've done this quite often over plain-jane unix TCP/IP sockets without issues. I had also thought that QDataStream's operator<< (and operator>>) functions were supposed to handle QImage transfers, but as shown above, all you get is a NULL image.

[wysota]

I had also thought that QDataStream's operator<< (and operator>>) functions were supposed to handle QImage transfers, but as shown above, all you get is a NULL image.

They don't handle any "transfers". If not all image data is available to read then you'll get a null image.

I don't see you wait for any data in the code you posted.

[KShots]

Ah, yes... I had not posted that part. I was calling a QTcpSocket->waitForReadyRead() before entering the readRawData() functions. That's why I get at least partial data.

I also just recently added more waitForReadyRead() calls inside the readRawData() function, and I now do in fact get the entire image.

As far as 'operator<<' and 'operator>>', are you suggesting that these are not suitable for use over sockets (meaning they'd work fine for QFile or similar)? Also, should I stop utilizing QDataStream::writeBytes() and QDataStream::readBytes() over TCP/IP for the same reasons? Am I just getting "lucky" that my XML content is small enough to be sent over in the first packet received?

[squidge]

You do understand that waitForReadyRead() blocks the current thread ? If this is your ui thread, it may cause the OS to terminate your code thinking it has become non-responsive.

Is there a reason you do not use the much more preferred signals and slots approach?

[wysota]

As far as 'operator<<' and 'operator>>', are you suggesting that these are not suitable for use over sockets (meaning they'd work fine for QFile or similar)? Also, should I stop utilizing QDataStream::writeBytes() and QDataStream::readBytes() over TCP/IP for the same reasons?

I'm only suggesting that they require complete data to be available. The source of the data is not relevant. If you'd like to read a rectangle but you have coordinates of only one point then no matter how hard you try, you can't create a rectangle out of it.

Am I just getting "lucky" that my XML content is small enough to be sent over in the first packet received?

Yes. Try congesting your link and you won't be so lucky anymore.

[KShots]

I do understand that, the socket is being utilized in a non-gui thread. I am using signals and slots to the extent that I can, but I seem to be missing my crystal ball for telling me whether I should be expecting XML content (possibly a fragment if I'm understanding the above correctly with QDataStream.readBytes possibly not filling in all the data required before returning), or a QImage fragment. I see two states: (1) Waiting for requests, and (2) processing a request. I don't see (3) resuming processing on an arbitrary fragmented request

[squidge]

Surely you would have a state machine which tells you what you are expecting and that all data comes with a header which states what is to follow and how many bytes are present in the payload. If you receive something you didn't expect you inform the sender and perform appropriate error recovery procedures. You would also have appropriate timeout mechanisms such that if you receive a partial response and it doesn't complete after such timeout, you reject the already received data, etc.

If you can receive the data using waitForReadyRead, then you can always receive the data using signals and slots (unless of course you have no event loop).

[KShots]

Surely you would have a state machine which tells you what you are expecting and that all data comes with a header which states what is to follow and how many bytes are present in the payload.

Yup, got that much... at least, all data comes with a header (in XML format), and even the header is prepended by a size variable telling me how large the header is... but regardless of how much data I expect to receive from what's in the header, that data just isn't going to be there (possibly the header itself is going to be fragmented) - so I need to read in what is available and wait for the rest to come in, and I need to do this outside of the event loop until this data is fully gathered (I can't operate on partial data). I can't see a means of waiting unless I call waitForReadyRead().

If you can receive the data using waitForReadyRead, then you can always receive the data using signals and slots (unless of course you have no event loop).

... how is this possible if your data is fragmented? I'm starting to feel like I'm missing some fundamental concept here...

[wysota]

That's simple:

class X : ... {
private slots:
  void moreDataArrived();
private:
  QByteArray buffer;
};
// ...
connect(socket, SIGNAL(readyRead()), this, SLOT(moreDataArrived()));
//...
void X::moreDataArrived() {
  QByteArray dat = socket->readAll(); // read all pending data
  buffer.append(dat); // and append it to your own buffer
 
  while(bufferContainsFullRequest(buffer)){ // if data incomplete then this won't execute
      MyRequestClass req = getRequest(buffer); // removes data from buffer
      processRequest(req);
  }
}

class X : ... {
private slots:
  void moreDataArrived();
private:
  QByteArray buffer;
};
// ...
connect(socket, SIGNAL(readyRead()), this, SLOT(moreDataArrived()));
//...
void X::moreDataArrived() {
  QByteArray dat = socket->readAll(); // read all pending data
  buffer.append(dat); // and append it to your own buffer

  while(bufferContainsFullRequest(buffer)){ // if data incomplete then this won't execute
      MyRequestClass req = getRequest(buffer); // removes data from buffer
      processRequest(req);
  }
}

To copy to clipboard, switch view to plain text mode