TCPSocket/Server

[TheGrimace]

I am making a simple file transfer app based on Threaded Fortune Server. I am running into an issue where I am losing the last couple of bytes from my file.

Client Code:

file.open(QIODevice::ReadOnly);
DataStream infile(&file);
DataStream in(socket);
int count = 0;
char point[1500];
while(!infile.atEnd())
		{
			count = infile.readRawData(point,1500);
			in.writeRawData(point,count);
			TCPSocket->waitForBytesWritten(300);
		}

file.open(QIODevice::ReadOnly);
DataStream infile(&file);
DataStream in(socket);
int count = 0;
char point[1500];
while(!infile.atEnd())
		{
			count = infile.readRawData(point,1500);
			in.writeRawData(point,count);
			TCPSocket->waitForBytesWritten(300);
		}

To copy to clipboard, switch view to plain text mode 

Server Side:
This function is connected to readyRead

void ReadNow{
QDataStream input(socket);
QDataStream output(file);
char point[1500];
int count = 0;
	while(!input.atEnd())
	{
		count = input.readRawData(point,1500);
		totalbytes+=count;
		output.writeRawData(point,count);
	}
	file.waitForBytesWritten(1000);
}

void ReadNow{
QDataStream input(socket);
QDataStream output(file);
char point[1500];
int count = 0;
	while(!input.atEnd())
	{
		count = input.readRawData(point,1500);
		totalbytes+=count;
		output.writeRawData(point,count);
	}
	file.waitForBytesWritten(1000);
}

To copy to clipboard, switch view to plain text mode 

I am unsure of where the missing bytes are going.

[marcel]

Oh, I see you posted some code .
The code seems ok.
Try comparing the file size(as reported by a file browser) with the size you read in the client and with totalbytes from the server.
Which is the wrong one?

Also, in the client, test the return values of writeRawData and waitForBytesWritten. Do they ever return -1?

Regards

[TheGrimace]

I just checked and it is wrong on the server side. Also worth noting: it never seems to get the full 1500 only about 1150. This may just be due to packetization, though. I am testing the -1 issue now.

Thank you for your quick responses Marcel. The reason i can post code this time is this is just a test app and not a company product.

[TheGrimace]

Darn! It is not returning -1. I am not sure where the missing bytes are.

[marcel]

Well, you either don't catch all the data on the server, or you don't send it all.

Do you use the QTcpSocket::readyRead signal? You should connect the ReadNow function to it and read all the data at once, not only 1500 bytes...

Anyway, sending ~1.5k of data and sleeping 300ms at most is not exactly optimal.
You could send larger quantities and QTcpSocket is buffered.

BTW: waitForBytesWritten returns false on timeout or error, not -1.

Regards

[TheGrimace]

I know I can send more, I just want to get it working before I accelerated it. I will check waitforbyteswritten now. Thank You.

[TheGrimace]

Just checked. It doesn't seem to return false either.

The ReadNow function is connected to readyRead. It is called about 400 (exageration) times.

[marcel]

Just for curiosity: remove(in the server) file.waitForBytesWritten.
There is no need for it, since it is buffered. At most you can call flush on it, but that is not needed either.

Regards

[TheGrimace]

Is there any reason that the readyread signal would not be raised at the very end of the data?

(Sorry about the triple post. I wish there were a way to edit forum posts on this site)

[TheGrimace]

No change after removing the waitForBytesWritten. The data is always lost at the end of the file and it is never the same amount. It varies between 1-10 K.

[marcel]

No, it should be emitted every time. And I am sure it is.

Regards

[TheGrimace]

I just tried comparing the bytes written to disk to the bytes received from the socket. They are the same. So that isn't it.

[TheGrimace]

Full Code: (just in case it is somewhere else) (btw I know this code is messy and much of it is not used. I was just making this app quick and for testing only)

Client:

MyClient::MyClient()
{
	actions = 0;
	startUDP = new QPushButton("StartUDP");
	startTCP = new QPushButton("StartTCP");
	Browse1 = new QPushButton("Browse");
	Browse2 = new QPushButton("Browse");
	portNumberUDP = new QLineEdit("");
	portNumberTCP = new QLineEdit("9654");
	TCPfile = new QLineEdit("");
	UDPfile = new QLineEdit("");
	nic = new QLineEdit("10.24.74.115");
	theLayout = new QGridLayout;
	lblStatusUDP = new QLabel;
	lblStatusTCP = new QLabel;
	theLayout->addWidget(portNumberUDP,0,2);
	theLayout->addWidget(portNumberTCP,1,2);
	theLayout->addWidget(Browse1, 0, 1);
	theLayout->addWidget(Browse2, 1, 1);
	theLayout->addWidget(startUDP,0,3);
	theLayout->addWidget(nic,0,4);
	theLayout->addWidget(TCPfile,1,0);
	theLayout->addWidget(UDPfile,0,0);
	theLayout->addWidget(startTCP,1,3);
	theLayout->addWidget(lblStatusUDP,2,0,1,3);
	theLayout->addWidget(lblStatusTCP,3,0,1,3);
	setLayout(theLayout);
 
	TCPSocket = new QTcpSocket();
 
	connect(startUDP, SIGNAL(clicked()), this, SLOT(requestAnotherUDP()));
	connect(startTCP, SIGNAL(clicked()), this, SLOT(requestAnotherTCP()));
	connect(Browse1,SIGNAL(clicked()),this,SLOT(catchBrowse1()));
	connect(Browse2,SIGNAL(clicked()),this,SLOT(catchBrowse2()));
	connect(TCPSocket, SIGNAL(connected()) , this , SLOT(catchConnected()));
}
void MyClient::requestAnotherUDP()
{
	actions++;
	QUdpSocket* UDPSocket = new QUdpSocket;
	QFile file(UDPfile->text());
	file.open(QIODevice::ReadOnly);
	QTextStream infile(&file);
 
	int sendData = qrand() % 10;
    startUDP->setEnabled(false);
    blockSize = 0;
	QByteArray block, BigBlock;
    UDPSocket->abort();
    UDPSocket->connectToHost(nic->text(),portNumberUDP->text().toInt());
	lblStatusUDP->setText("UDP: " + QString::number(actions) + ". Sending file to "+nic->text()+" on port " + portNumberUDP->text());
	QDataStream in(UDPSocket);
	in.setVersion(QDataStream::Qt_4_0);
	while(!infile.atEnd())
	{
		infile >> block;
		block.append(' ');
		BigBlock.append(block);
	}
	in.writeBytes(BigBlock,BigBlock.length());
	UDPSocket->disconnect();
	UDPSocket->waitForDisconnected(3000);
	lblStatusUDP->setText("UDP: " + QString::number(actions) + ". Sent file to "+nic->text()+" on port " + portNumberUDP->text());
    startUDP->setEnabled(true);
}
void MyClient::requestAnotherTCP()
{
	actions++;
    startTCP->setEnabled(false);
    blockSize = 0;
	TCPSocket->abort();
	TCPSocket->connectToHost(nic->text(),portNumberTCP->text().toInt());
	TCPSocket->setReadBufferSize(50000000);
}
 
void MyClient::catchConnected()
{
	qDebug("Connected");
 
	QByteArray block, BigBlock;
	QFile file(TCPfile->text());
	file.open(QIODevice::ReadOnly);
	QDataStream infile(&file);
	char point[1500];
 
    if(TCPSocket->isValid())
	{
		lblStatusTCP->setText("TCP: " + QString::number(actions) + ". Sending file to "+nic->text()+" on port " + portNumberTCP->text());
		QDataStream in(TCPSocket);
		in.setVersion(QDataStream::Qt_4_3);
		QFile newfile("Test.dat");
		newfile.open(QIODevice::WriteOnly);
		QDataStream fi(&newfile);
		int totalbytes = 0;
		int totalbytes2 = 0;
		int count = -1;
		int count2 = -1;
		while(!infile.atEnd())
		{
			count = infile.readRawData(point,1500);
			count2 = in.writeRawData(point,count);
			if(!TCPSocket->waitForBytesWritten(300))
			{
				QMessageBox::about(this,"False","False");
			}
			totalbytes2 += fi.writeRawData(point,count);
			if(count2 == -1)
			{
				QMessageBox::about(this,"-1","-1");
			}
		}
		TCPSocket->waitForBytesWritten(3000);
		newfile.waitForBytesWritten(3000);
		newfile.close();
 
		startTCP->setEnabled(true);
		TCPSocket->flush();
		TCPSocket->disconnectFromHost();
		TCPSocket->waitForDisconnected(-1);
		lblStatusTCP->setText("TCP: " + QString::number(actions) + ". Sent file to "+nic->text()+" on port " + portNumberTCP->text());
	}
 
 
}
void MyClient::catchBrowse1()
{
	QString strfile = QFileDialog::getOpenFileName(this,"UDP File");
 
	if(!strfile.isEmpty())
	{
		UDPfile->setText(strfile);
	}
}
void MyClient::catchBrowse2()
{
	QString strfile = QFileDialog::getOpenFileName(this,"TCP File");
 
	if(!strfile.isEmpty())
	{
		TCPfile->setText(strfile);
	}
}

MyClient::MyClient()
{
	actions = 0;
	startUDP = new QPushButton("StartUDP");
	startTCP = new QPushButton("StartTCP");
	Browse1 = new QPushButton("Browse");
	Browse2 = new QPushButton("Browse");
	portNumberUDP = new QLineEdit("");
	portNumberTCP = new QLineEdit("9654");
	TCPfile = new QLineEdit("");
	UDPfile = new QLineEdit("");
	nic = new QLineEdit("10.24.74.115");
	theLayout = new QGridLayout;
	lblStatusUDP = new QLabel;
	lblStatusTCP = new QLabel;
	theLayout->addWidget(portNumberUDP,0,2);
	theLayout->addWidget(portNumberTCP,1,2);
	theLayout->addWidget(Browse1, 0, 1);
	theLayout->addWidget(Browse2, 1, 1);
	theLayout->addWidget(startUDP,0,3);
	theLayout->addWidget(nic,0,4);
	theLayout->addWidget(TCPfile,1,0);
	theLayout->addWidget(UDPfile,0,0);
	theLayout->addWidget(startTCP,1,3);
	theLayout->addWidget(lblStatusUDP,2,0,1,3);
	theLayout->addWidget(lblStatusTCP,3,0,1,3);
	setLayout(theLayout);

	TCPSocket = new QTcpSocket();

	connect(startUDP, SIGNAL(clicked()), this, SLOT(requestAnotherUDP()));
	connect(startTCP, SIGNAL(clicked()), this, SLOT(requestAnotherTCP()));
	connect(Browse1,SIGNAL(clicked()),this,SLOT(catchBrowse1()));
	connect(Browse2,SIGNAL(clicked()),this,SLOT(catchBrowse2()));
	connect(TCPSocket, SIGNAL(connected()) , this , SLOT(catchConnected()));
}
void MyClient::requestAnotherUDP()
{
	actions++;
	QUdpSocket* UDPSocket = new QUdpSocket;
	QFile file(UDPfile->text());
	file.open(QIODevice::ReadOnly);
	QTextStream infile(&file);

	int sendData = qrand() % 10;
    startUDP->setEnabled(false);
    blockSize = 0;
	QByteArray block, BigBlock;
    UDPSocket->abort();
    UDPSocket->connectToHost(nic->text(),portNumberUDP->text().toInt());
	lblStatusUDP->setText("UDP: " + QString::number(actions) + ". Sending file to "+nic->text()+" on port " + portNumberUDP->text());
	QDataStream in(UDPSocket);
	in.setVersion(QDataStream::Qt_4_0);
	while(!infile.atEnd())
	{
		infile >> block;
		block.append(' ');
		BigBlock.append(block);
	}
	in.writeBytes(BigBlock,BigBlock.length());
	UDPSocket->disconnect();
	UDPSocket->waitForDisconnected(3000);
	lblStatusUDP->setText("UDP: " + QString::number(actions) + ". Sent file to "+nic->text()+" on port " + portNumberUDP->text());
    startUDP->setEnabled(true);
}
void MyClient::requestAnotherTCP()
{
	actions++;
    startTCP->setEnabled(false);
    blockSize = 0;
	TCPSocket->abort();
	TCPSocket->connectToHost(nic->text(),portNumberTCP->text().toInt());
	TCPSocket->setReadBufferSize(50000000);
}

void MyClient::catchConnected()
{
	qDebug("Connected");

	QByteArray block, BigBlock;
	QFile file(TCPfile->text());
	file.open(QIODevice::ReadOnly);
	QDataStream infile(&file);
	char point[1500];

    if(TCPSocket->isValid())
	{
		lblStatusTCP->setText("TCP: " + QString::number(actions) + ". Sending file to "+nic->text()+" on port " + portNumberTCP->text());
		QDataStream in(TCPSocket);
		in.setVersion(QDataStream::Qt_4_3);
		QFile newfile("Test.dat");
		newfile.open(QIODevice::WriteOnly);
		QDataStream fi(&newfile);
		int totalbytes = 0;
		int totalbytes2 = 0;
		int count = -1;
		int count2 = -1;
		while(!infile.atEnd())
		{
			count = infile.readRawData(point,1500);
			count2 = in.writeRawData(point,count);
			if(!TCPSocket->waitForBytesWritten(300))
			{
				QMessageBox::about(this,"False","False");
			}
			totalbytes2 += fi.writeRawData(point,count);
			if(count2 == -1)
			{
				QMessageBox::about(this,"-1","-1");
			}
		}
		TCPSocket->waitForBytesWritten(3000);
		newfile.waitForBytesWritten(3000);
		newfile.close();

		startTCP->setEnabled(true);
		TCPSocket->flush();
		TCPSocket->disconnectFromHost();
		TCPSocket->waitForDisconnected(-1);
		lblStatusTCP->setText("TCP: " + QString::number(actions) + ". Sent file to "+nic->text()+" on port " + portNumberTCP->text());
	}

	
}
void MyClient::catchBrowse1()
{
	QString strfile = QFileDialog::getOpenFileName(this,"UDP File");
	
	if(!strfile.isEmpty())
	{
		UDPfile->setText(strfile);
	}
}
void MyClient::catchBrowse2()
{
	QString strfile = QFileDialog::getOpenFileName(this,"TCP File");
	
	if(!strfile.isEmpty())
	{
		TCPfile->setText(strfile);
	}
}

To copy to clipboard, switch view to plain text mode 

Server:

#include "testserver.h"
 
tcpServer::tcpServer(): QWidget()
{
	node = 1;
	TCPServ = new QTcpServer;
	connect(TCPServ,SIGNAL(newConnection()),this,SLOT(createNewSocket()));
	txtEdit = new QTextEdit;
	QGridLayout* theLayout = new QGridLayout;
	port = new QLineEdit;
	nic = new QComboBox;
	start = new QPushButton;
	start->setText("Start");
	start->setCheckable(true);
	connect(start,SIGNAL(clicked()),this,SLOT(catchstart()));
	theLayout->addWidget(port,0,0);
	theLayout->addWidget(nic,0,1);
	theLayout->addWidget(start,0,2);
	theLayout->addWidget(txtEdit,1,0,1,3);
	port->setText("port");
 
	QList<QHostAddress> nicList = QNetworkInterface::allAddresses();
	for(int i = nicList.size()-1; i>=0; i--)
	{
		this->nic->insertItem(i, nicList[i].toString());
	}
	setLayout(theLayout);
}
void tcpServer::createNewSocket()
{
	QTcpSocket * thesocket = TCPServ->nextPendingConnection();
	ServerNode* newNode = new ServerNode(this,thesocket,this->txtEdit, node++);
 
}
void tcpServer::catchstart()
{
	if(start->isChecked())
	{
		start->setText("Stop");
		TCPServ->listen(QHostAddress(nic->currentText()),port->text().toInt());
	}
	else
	{
		start->setText("Start");
		TCPServ->close();
	}
}
void tcpServer::changeText(QString newstring)
{
	txtEdit->append("\n"+newstring);
}
ServerNode::ServerNode()
{
}
ServerNode::ServerNode(tcpServer* parent,QTcpSocket* thesocket,QTextEdit* point, int intnode)
{
	node = intnode;
	totalbytes = 0;
	connect(thesocket,SIGNAL(readyRead()),this,SLOT(readNow()));
	//connect(thesocket,SIGNAL(disconnected()),this,SLOT(fileDone()));
	input.setDevice(thesocket);
	socket = thesocket;
	file .setFileName(QString("Test_%1").arg(node));
	file.open(QIODevice::Append);
	output.setDevice(&file);
}
void ServerNode::readNow()
{
	int count = 0;
	int count2 = 0;
	char point[1500];
 
	QByteArray oneBlock;
 
	while(!input.atEnd())
	{
		count = input.readRawData(point,1500);
		if(count == -1)
		{
				QMessageBox::about(0,"-1","-1");
		}
		totalbytes+=count;
		count2 = output.writeRawData(point,count);
		if(count != count2)
		{
			QMessageBox::about(0,"1!=2","1!=2");
		}
	}
	//file.waitForBytesWritten(1000);
}
void ServerNode::run()
{
}
void ServerNode::fileDone()
{
	QFile::rename(QString("Temp__%1.dat").arg(node),QString("Complete__%1.dat").arg(node));
}

#include "testserver.h"

tcpServer::tcpServer(): QWidget()
{
	node = 1;
	TCPServ = new QTcpServer;
	connect(TCPServ,SIGNAL(newConnection()),this,SLOT(createNewSocket()));
	txtEdit = new QTextEdit;
	QGridLayout* theLayout = new QGridLayout;
	port = new QLineEdit;
	nic = new QComboBox;
	start = new QPushButton;
	start->setText("Start");
	start->setCheckable(true);
	connect(start,SIGNAL(clicked()),this,SLOT(catchstart()));
	theLayout->addWidget(port,0,0);
	theLayout->addWidget(nic,0,1);
	theLayout->addWidget(start,0,2);
	theLayout->addWidget(txtEdit,1,0,1,3);
	port->setText("port");
	
	QList<QHostAddress> nicList = QNetworkInterface::allAddresses();
	for(int i = nicList.size()-1; i>=0; i--)
	{
		this->nic->insertItem(i, nicList[i].toString());
	}
	setLayout(theLayout);
}
void tcpServer::createNewSocket()
{
	QTcpSocket * thesocket = TCPServ->nextPendingConnection();
	ServerNode* newNode = new ServerNode(this,thesocket,this->txtEdit, node++);
	
}
void tcpServer::catchstart()
{
	if(start->isChecked())
	{
		start->setText("Stop");
		TCPServ->listen(QHostAddress(nic->currentText()),port->text().toInt());
	}
	else
	{
		start->setText("Start");
		TCPServ->close();
	}
}
void tcpServer::changeText(QString newstring)
{
	txtEdit->append("\n"+newstring);
}
ServerNode::ServerNode()
{
}
ServerNode::ServerNode(tcpServer* parent,QTcpSocket* thesocket,QTextEdit* point, int intnode)
{
	node = intnode;
	totalbytes = 0;
	connect(thesocket,SIGNAL(readyRead()),this,SLOT(readNow()));
	//connect(thesocket,SIGNAL(disconnected()),this,SLOT(fileDone()));
	input.setDevice(thesocket);
	socket = thesocket;
	file .setFileName(QString("Test_%1").arg(node));
	file.open(QIODevice::Append);
	output.setDevice(&file);
}
void ServerNode::readNow()
{
	int count = 0;
	int count2 = 0;
	char point[1500];

	QByteArray oneBlock;

	while(!input.atEnd())
	{
		count = input.readRawData(point,1500);
		if(count == -1)
		{
				QMessageBox::about(0,"-1","-1");
		}
		totalbytes+=count;
		count2 = output.writeRawData(point,count);
		if(count != count2)
		{
			QMessageBox::about(0,"1!=2","1!=2");
		}
	}
	//file.waitForBytesWritten(1000);
}
void ServerNode::run()
{
}
void ServerNode::fileDone()
{
	QFile::rename(QString("Temp__%1.dat").arg(node),QString("Complete__%1.dat").arg(node));
}

To copy to clipboard, switch view to plain text mode 

[marcel]

In readNow, try reading all the available data at once, without the data stream wrapped around the socket.
Use the QIODevice API directly.
If it works, then it means there is a problem in the data stream handling.

Regards

[TheGrimace]

I tried changing the readrawdata command in both the server and the client socket writes/reads to write and read commands directly on the socket. This does not seem to have made a difference.

[marcel]

OK. Weird enough.
I'm gonna write a small sample( still based on the fortune server/client) to see what's wrong.

I'll post back when its done.

regards

[TheGrimace]

Thank you!

Just so you know I only have the problem with files over a meg or so in size.

[TheGrimace]

I will not be able to check this forum until monday. Thank you for all the help you have given me so far. It is refreshing to know that the mistake was not an obvious one.

Again, Thank You,

The Grimace

[marcel]

Well, I couldn't find anything.
I modified the samples and attached them. Here the server sends the file. I used an 8.42mb file and a 15.6mb one, and I couldn't reproduce the problem.

You can test on your side.
The send/receive methods are similar to yours. I also used QDataStream's.

Everything happens in Client::readFortune(receive file) and in FortuneThread::run( sends the file).

The file name is hardcoded, so you might want to modify that.

Regards

[TheGrimace]

Your code worked for me as well. I am still not sure what the issue was, but I was able to integrate your code into mine to get the app working.

Thank you very much for all your help.