File transfer through sockets

[p3c0]

I want to send a large file using qt's socket programming for this is edited the existing fortune client-server code as follows:

Server code:-

void Server::SendData()
{
QByteArray block;
QByteArray data;
QDataStream out(&block, QIODevice::WriteOnly);
out.setVersion(QDataStream::Qt_4_0);
while(1)
{
out << quint64(0);
data = inputFile.read(32768*8);
qDebug() << "Read : " << data.size();
if(data.size()==0)
break;
out << data;
out.device()->seek(0);
out << quint64(block.size() - sizeof(quint64));
client.write(block);
qDebug() << "Written : " << block.size();
//block.clear();
}
client.disconnectFromHost();
client.waitForDisconnected();
}

Client Code:-

void Client::ReadData()
{
QFile file(filename);
if(!(file.open(QIODevice::Append)))
{
qDebug("File cannot be opened.");
exit(0);
}

QDataStream in(client);
QByteArray data;
in.setVersion(QDataStream::Qt_4_0);


if (blockSize == 0) {
if (client->bytesAvailable() < sizeof(quint64))
return;
in >> blockSize;
}

if (client->bytesAvailable() < blockSize)
return;

data = client->readAll();
file.write(data);
qDebug() << "Written : " << data.size() ;
blockSize = 0;
file.close();
}

So according to documentation whenever data is available on socket the readyRead() signal is emitted, i have connected Slot ReadData() to it.
The problem is full file is not transferred on the client side ,lot of data is missing(tested 1.3 G, only 1.2 G was received)
Can any one please tell what is wrong in the above code?

[Surendil]

May server transfer blocks with length less than client expects? I can be mistaken, but last block will not be saved.

if (client->bytesAvailable() < blockSize)
return;

if (client->bytesAvailable() < blockSize)
return;

To copy to clipboard, switch view to plain text mode 

Try to find block on which transferring stops and check client's behaviour.

[p3c0]

The server reads 262144 bytes(which is guess is correct according to size specified i.e 32768*8)from file and writes 262164 bytes to socket. The extra 20 bytes are of the size of the block i think.
....
Read : 262144
Written : 262164
....

The clients receives block of size 671782 bytes on an average.

....
Written : 671782
Written : 622650
Written : 671782
....

so according to me the server writes all the blocks to the socket, but clients misses out some.

[wysota]

You are falsely assuming that the fortune cookie example is fit for every possible purpose of using TCP. In your current code if you read the block size of say... 20 bytes and there is 4GB of data available in the socket, then you read all those 4GB, completely ignoring the block size. Next time you try to read the block size again but you are at a completely arbitrary point of the stream so you are misinterpreting the block size and discarding those four bytes (and some more too because then you are falsely assuming there is a QByteArray object waiting for you in the socket). The fact that you send data in one chunk doesn't mean it will arrive in one chunk, in most cases it will not (unless you trully believe you can push `.3GB of data into a single TCP segment). Why are you using QDataStream here at all? Is that a concious decision?

Please search the forum, this issue has been discussed countless times.

[p3c0]

Well solved the problem.. going by the traditional way of reading chunks of data and writing to it.
Here's the full code:

Server Code:
----------------------
Server.cpp

#include "server.h"
#include <QtNetwork>
 
Server::Server(QObject *parent) :
    QTcpServer(parent)
{
    //connect(&server, SIGNAL(newConnection()),this, SLOT(acceptConnection()));
    listen(QHostAddress::Any, 8888);
    qDebug() << serverAddress() << serverPort() << serverError();
}
 
Server::~Server()
{
    close();
}
 
void Server::incomingConnection(int socketDescriptor)
{
    qDebug() << "File transfer started";
 
    SendThread *thread = new SendThread(socketDescriptor,this);
    connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
    thread->start();
    qDebug() << "Thread called";
}

#include "server.h"
#include <QtNetwork>

Server::Server(QObject *parent) :
    QTcpServer(parent)
{
    //connect(&server, SIGNAL(newConnection()),this, SLOT(acceptConnection()));
    listen(QHostAddress::Any, 8888);
    qDebug() << serverAddress() << serverPort() << serverError();
}

Server::~Server()
{
    close();
}

void Server::incomingConnection(int socketDescriptor)
{
    qDebug() << "File transfer started";

    SendThread *thread = new SendThread(socketDescriptor,this);
    connect(thread, SIGNAL(finished()), thread, SLOT(deleteLater()));
    thread->start();
    qDebug() << "Thread called";
}

To copy to clipboard, switch view to plain text mode 

server.h

#ifndef SERVER_H
#define SERVER_H
 
#include <QObject>
#include <QDebug>
#include <QTcpServer>
#include "sendthread.h"
 
class Server : public QTcpServer
{
    Q_OBJECT
 
public:
    explicit Server(QObject *parent = 0);
    void ServerListener();
    ~Server();
 
protected:
    void incomingConnection(int socketDescriptor);
 
signals:
 
public slots:
 
};
 
#endif // SERVER_H

#ifndef SERVER_H
#define SERVER_H

#include <QObject>
#include <QDebug>
#include <QTcpServer>
#include "sendthread.h"

class Server : public QTcpServer
{
    Q_OBJECT

public:
    explicit Server(QObject *parent = 0);
    void ServerListener();
    ~Server();

protected:
    void incomingConnection(int socketDescriptor);

signals:

public slots:

};

#endif // SERVER_H

To copy to clipboard, switch view to plain text mode 

SendThread.cpp

#include "sendthread.h"
#define FILENAME "/root/Tars/Qt_SDK_Win_offline_v1_2_en.exe"
 
SendThread::SendThread(int socketDescriptor,QObject *parent)
    : QThread(parent), socketDescriptor(socketDescriptor)
{
}
 
void SendThread::run()
{
    QTcpSocket client;
    qDebug() << "Thread Descriptor :" << socketDescriptor;
    if (!client.setSocketDescriptor(socketDescriptor))
    {
        qDebug() << client.error();
        return;
    }
    qDebug() << "Thread : Connected";
 
    //send File
    QFile inputFile(FILENAME);
    QByteArray read;
    inputFile.open(QIODevice::ReadOnly);
    while(1)
    {
        read.clear();
        read = inputFile.read(32768*8);
        qDebug() << "Read : " << read.size();
        if(read.size()==0)
            break;
 
        qDebug() << "Written : " << client.write(read);
        client.waitForBytesWritten();
        read.clear();
    }
    inputFile.close();
    client.disconnectFromHost();
    client.waitForDisconnected();
    qDebug() << "Thread : File transfer completed";
}

#include "sendthread.h"
#define FILENAME "/root/Tars/Qt_SDK_Win_offline_v1_2_en.exe"

SendThread::SendThread(int socketDescriptor,QObject *parent)
    : QThread(parent), socketDescriptor(socketDescriptor)
{
}

void SendThread::run()
{
    QTcpSocket client;
    qDebug() << "Thread Descriptor :" << socketDescriptor;
    if (!client.setSocketDescriptor(socketDescriptor))
    {
        qDebug() << client.error();
        return;
    }
    qDebug() << "Thread : Connected";

    //send File
    QFile inputFile(FILENAME);
    QByteArray read;
    inputFile.open(QIODevice::ReadOnly);
    while(1)
    {
        read.clear();
        read = inputFile.read(32768*8);
        qDebug() << "Read : " << read.size();
        if(read.size()==0)
            break;

        qDebug() << "Written : " << client.write(read);
        client.waitForBytesWritten();
        read.clear();
    }
    inputFile.close();
    client.disconnectFromHost();
    client.waitForDisconnected();
    qDebug() << "Thread : File transfer completed";
}

To copy to clipboard, switch view to plain text mode 

SendThread.h

#ifndef SENDTHREAD_H
#define SENDTHREAD_H
#include <QThread>
#include <QFile>
#include <QTcpSocket>
 
class SendThread : public QThread
{
public:
    SendThread(int socketdescriptor,QObject *parent);
    void run();
 
private:
    int socketDescriptor;
};
 
#endif // SENDTHREAD_H

#ifndef SENDTHREAD_H
#define SENDTHREAD_H
#include <QThread>
#include <QFile>
#include <QTcpSocket>

class SendThread : public QThread
{
public:
    SendThread(int socketdescriptor,QObject *parent);
    void run();

private:
    int socketDescriptor;
};

#endif // SENDTHREAD_H

To copy to clipboard, switch view to plain text mode 

Client Code:-
------------------------

Client.cpp

#include "client.h"
 
Client::Client(QObject *parent) :
    QObject(parent)
{
    client = new QTcpSocket(this);
    client->abort();
    connect(client, SIGNAL(readyRead()), this, SLOT(ReadData()));
    connect(client, SIGNAL(disconnected()), this, SLOT(Completed()));
 
}
 
Client::~Client()
{
    client->close();
}
 
void Client::start(QString address, quint16 port, QString file)
{
    QHostAddress addr(address);
    filename = file;
    client->connectToHost(addr, port);
    qDebug() << client->socketDescriptor();
}
 
 
void Client::Completed()
{
    qDebug() << "File transfer complete";
}
 
void Client::ReadData()
{
    QFile file(filename);
    if(!(file.open(QIODevice::Append)))
    {
        qDebug("File cannot be opened.");
        exit(0);
    }
    QByteArray read = client->read(client->bytesAvailable());
    qDebug() << "Read    : " << read.size();
    file.write(read);
    file.close();
}

#include "client.h"

Client::Client(QObject *parent) :
    QObject(parent)
{
    client = new QTcpSocket(this);
    client->abort();
    connect(client, SIGNAL(readyRead()), this, SLOT(ReadData()));
    connect(client, SIGNAL(disconnected()), this, SLOT(Completed()));

}

Client::~Client()
{
    client->close();
}

void Client::start(QString address, quint16 port, QString file)
{
    QHostAddress addr(address);
    filename = file;
    client->connectToHost(addr, port);
    qDebug() << client->socketDescriptor();
}


void Client::Completed()
{
    qDebug() << "File transfer complete";
}

void Client::ReadData()
{
    QFile file(filename);
    if(!(file.open(QIODevice::Append)))
    {
        qDebug("File cannot be opened.");
        exit(0);
    }
    QByteArray read = client->read(client->bytesAvailable());
    qDebug() << "Read    : " << read.size();
    file.write(read);
    file.close();
}

To copy to clipboard, switch view to plain text mode 

client.h

#ifndef CLIENT_H
#define CLIENT_H
 
#include <QObject>
#include <QTcpSocket>
#include <QHostAddress>
#include <QFile>
 
class Client : public QObject
{
    Q_OBJECT
public:
    explicit Client(QObject *parent = 0);
    ~Client();
    QTcpSocket *client;
    void start(QString address, quint16 port, QString file);
    QString filename;
 
private:
 
signals:
 
public slots:
    void ReadData();
    void Completed();
};
 
#endif // CLIENT_H

#ifndef CLIENT_H
#define CLIENT_H

#include <QObject>
#include <QTcpSocket>
#include <QHostAddress>
#include <QFile>

class Client : public QObject
{
    Q_OBJECT
public:
    explicit Client(QObject *parent = 0);
    ~Client();
    QTcpSocket *client;
    void start(QString address, quint16 port, QString file);
    QString filename;

private:

signals:

public slots:
    void ReadData();
    void Completed();
};

#endif // CLIENT_H

To copy to clipboard, switch view to plain text mode 

Call Client as:

Client c;
c.start("192.168.10.210",8888,"/root/file1");

[wysota]

Oh, great... yet another one using threads for networking... /me facepalms


By the way, if you want to transfer a file between two hosts, why don't you use something called "file transfer protocol"?

[Quenix]

Thank you very much for your example. It helps me a lot to introduce me to QT code and TCP data transfer too. (I'm a student)

I tried your code for a basic project at school : transfer a movie from a server to a client and play it on the client side.
I just have to made some modification to your code in order to make it work properly.

1. I had to write a waitForReadyRead in the client side because it doesn't do anything when the SIGNAL(readyRead()) happens

void Client::start(QString address, quint16 port, QString file)
{
 
    QHostAddress addr(address);
    filename = file;
    client->connectToHost(addr, port);
    client->waitForReadyRead(); // this one have to be added

void Client::start(QString address, quint16 port, QString file)
{

    QHostAddress addr(address);
    filename = file;
    client->connectToHost(addr, port);
    client->waitForReadyRead(); // this one have to be added

To copy to clipboard, switch view to plain text mode 

And, in real life, data can be much longer than we think and the server can send more than juste one packet, so the client have to read until there's no data in the Socket :
So I just rearrange your code this way :

bool Client::ClientReadData()
{
    bool retour;
    QFile file(filename);
    QByteArray read;
    if(!(file.open(QIODevice::Append)))
    {
        qDebug("File cannot be opened.");
        exit(0);
    }
    else
    {
        while(client->waitForReadyRead())
        {
            read.clear();
            read = client->read(client->bytesAvailable());
            qDebug() << "Read    : " << read.size();
            qDebug() << "Written : " << file.write(read);
        }
    }
 
 
    file.close();
    if(client->waitForReadyRead() == true)
    {
        qDebug() << "Status : OK";
        retour = true;
    }
    else
    {
        qDebug() << "Status : Error : there's always data on the Socket when it disconected";
        retour = false;
    }
    return retour;
}

bool Client::ClientReadData()
{
    bool retour;
    QFile file(filename);
    QByteArray read;
    if(!(file.open(QIODevice::Append)))
    {
        qDebug("File cannot be opened.");
        exit(0);
    }
    else
    {
        while(client->waitForReadyRead())
        {
            read.clear();
            read = client->read(client->bytesAvailable());
            qDebug() << "Read    : " << read.size();
            qDebug() << "Written : " << file.write(read);
        }
    }


    file.close();
    if(client->waitForReadyRead() == true)
    {
        qDebug() << "Status : OK";
        retour = true;
    }
    else
    {
        qDebug() << "Status : Error : there's always data on the Socket when it disconected";
        retour = false;
    }
    return retour;
}

To copy to clipboard, switch view to plain text mode 

Again, thank you very much for your example and your code.

Serge

[anda_skoa]

Thank you very much for your example. It helps me a lot to introduce me to QT code and TCP data transfer too. (I'm a student)

I tried your code for a basic project at school : transfer a movie from a server to a client and play it on the client side.
I just have to made some modification to your code in order to make it work properly.

1. I had to write a waitForReadyRead in the client side because it doesn't do anything when the SIGNAL(readyRead()) happens

No. The client example code was even driven, no need to call any waitForXYZ. Whenever something happens on the socket, e.g. data received, a signal will be emitted and the client code reacts to that. In the example code it reads the available data and appends it to the file.

Your modification effetively blocks the ClientReadData() method until all data has been transferred, totally blocking the thread that executes it, needlessly introducing the need for a thread in order not to block the main thread.

There is no need for threading here at all, it just complicates the two programs and makes them harder to debug.

Cheers,
_

[Quenix]

No. The client example code was even driven, no need to call any waitForXYZ. Whenever something happens on the socket, e.g. data received, a signal will be emitted and the client code reacts to that. In the example code it reads the available data and appends it to the file.

Thank you very much for your answer.

What should I use instead of a client->waitForSomething because without this, nothing happens in the client side after the initial TCP Handshake when the server started to write in the socket.

Your modification effetively blocks the ClientReadData() method until all data has been transferred, totally blocking the thread that executes it, needlessly introducing the need for a thread in order not to block the main thread.

There is no need for threading here at all, it just complicates the two programs and makes them harder to debug.

The only way I see to do it better is to introduce a new threads, but that's because I'm a beginner ;-)
How could it be written in a better way ?

Thank you,

--
Serge

[anda_skoa]

Check that you have started the main thread's event loop, i.e. called exec() on the application object.

Cheers,
_

[Quenix]

Check that you have started the main thread's event loop, i.e. called exec() on the application object.

I tried the code in a QT application with GUI and I create the Class Client object with a button.clicked(). When I click on the button, I thought that the QApplication::exec() happens in the main.cpp before I pressed on the button. So, normally, the main thread's event loop is supposed to be started before I click anywhere on the mainwindow .?

Thank you very much for the time you take to answer all my newbe question :-)

--
Serge

[anda_skoa]

Yes, if you have a working GUI then the eventloop should be running.

Hard to tell without a minimal program using the concrete code.
All signal/slots setup correctly?

Cheers,
_

[Quenix]

All signal/slots setup correctly?

This is exactly the code you can see upper.
For the rest, there's just 1 button in the client side and I used "Go to slot..." in qtdesigner to create the SIGNAL/SLOT connection.

thanks,
Serge

[anda_skoa]

Well, since you claim this is all the code and the code does not contain any signal/slot connects, you are obviously missing those.

Cheers,
_