Putting large size file on FTP using QNetworkAccessManager

[darshan.hardas]

We are facing issues when we are trying to send the file (text/image/exe) to the FTP server.

We have used QFtp functions to put the file onto the server with user authentication. But QFtp is taking huge time to upload the files. Hence we moved to QNetworkAccessManager an have used the put method for uploading.

QNetworkAccessManager::put() returns with a reply and finished is also triggered, but no file is getting uploaded.

I have used the below code to put:

QUrl uploadurl("ftp://server/dir/");
    uploadurl.setUserName("XXXXXX");
    uploadurl.setPassword("XXXXXX");
    uploadurl.setPort(21);
    QNetworkRequest upload(uploadurl);
    QNetworkAccessManager *uploadman = new QNetworkAccessManager(this);
 
    QFile* file = new QFile("filepath");
    if(file->open(QFile::ReadOnly)) {
         uploadman->put(upload, file);
    }

QUrl uploadurl("ftp://server/dir/");
    uploadurl.setUserName("XXXXXX");
    uploadurl.setPassword("XXXXXX");
    uploadurl.setPort(21);
    QNetworkRequest upload(uploadurl);
    QNetworkAccessManager *uploadman = new QNetworkAccessManager(this);

    QFile* file = new QFile("filepath");
    if(file->open(QFile::ReadOnly)) {
         uploadman->put(upload, file);
    }

To copy to clipboard, switch view to plain text mode 

Please help us out with the above issue. Any example code would be a great help.

Thanks in advance.

[ChrisW67]

Given that you are discarding the QNetworkReply* returned by put(), and not connecting the QNetworkAccessManager instance to anything I do not see how you can possibly know that finished() is being emitted and that no error is being returned.


This code uploads just fine when the server permits it (and prints a non-zero error number if not):

#include <QtCore>
#include <QtNetwork>
#include <QDebug>
 
class Uploader: public QObject
{
    Q_OBJECT
public:
    Uploader(QObject *p = 0): QObject(p)     {  }
 
    void start(const QString &file)     {
        QUrl url("ftp://localhost/incoming");
        url.setUserName("test");
        url.setPassword("password");
 
        data = new QFile(file, this);
        if (data->open(QIODevice::ReadOnly)) {
            reply = nam.put(QNetworkRequest(url), data);
            connect(reply, SIGNAL(uploadProgress(qint64, qint64)), SLOT(uploadProgress(qint64, qint64)));
            connect(reply, SIGNAL(finished()), SLOT(uploadDone()));
        }   
        else 
            qDebug() << "Oops";
    }
 
public slots:
    void uploadProgress(qint64 bytesSent, qint64 bytesTotal)    {
        qDebug() << "Uploaded" << bytesSent << "of" << bytesTotal;
    }
 
    void uploadDone()     {
        qDebug() << "Finished" << reply->error();
        data->deleteLater();
        reply->deleteLater();
    }
 
private:
    QNetworkAccessManager nam;
    QFile *data;
    QNetworkReply *reply;
};
 
int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);
    Uploader u;
    u.start("ten_meg_file.dat");
    return app.exec();
}
#include "main.moc"

#include <QtCore>
#include <QtNetwork>
#include <QDebug>

class Uploader: public QObject
{
    Q_OBJECT
public:
    Uploader(QObject *p = 0): QObject(p)     {  }

    void start(const QString &file)     {
        QUrl url("ftp://localhost/incoming");
        url.setUserName("test");
        url.setPassword("password");

        data = new QFile(file, this);
        if (data->open(QIODevice::ReadOnly)) {
            reply = nam.put(QNetworkRequest(url), data);
            connect(reply, SIGNAL(uploadProgress(qint64, qint64)), SLOT(uploadProgress(qint64, qint64)));
            connect(reply, SIGNAL(finished()), SLOT(uploadDone()));
        }   
        else 
            qDebug() << "Oops";
    }

public slots:
    void uploadProgress(qint64 bytesSent, qint64 bytesTotal)    {
        qDebug() << "Uploaded" << bytesSent << "of" << bytesTotal;
    }

    void uploadDone()     {
        qDebug() << "Finished" << reply->error();
        data->deleteLater();
        reply->deleteLater();
    }

private:
    QNetworkAccessManager nam;
    QFile *data;
    QNetworkReply *reply;
};

int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);
    Uploader u;
    u.start("ten_meg_file.dat");
    return app.exec();
}
#include "main.moc"

To copy to clipboard, switch view to plain text mode 

[darshan.hardas]

Thanks for the input. I have not appended the file name to be uploaded in the QUrl formed. Now I can send the file to the FTP.

I am not getting the speed while the FTP is done. If upload/download speed is compared with the other FTP clients like FileZilla, the file is taking too much of time. The network speed of the machine used is in some mbps still the speed is very slow for upload using QFTP or QNetworkAccessManager .

Please let me know your valuable inputs.

[ChrisW67]

How do you expect us to debug your network connection? How big is the file? What speed is the connection? How long does it actually take? What else is your program doing?

[darshan.hardas]

The logic is implemented to put the file onto the shared FTP folder.

I have used QFtp and QNetworkAccessManger to put the file to the ftp. But the time taking to upload is very large. Compared to other FTP clients the time taken is just double the time taken.


I have used the files upto 100 mb. For 100 Mb the time taken is approx 12 mins by QFTP/QNetworkAcessManager. But the ftp clients like FileZilla uploads the file in 3-4 mins.

I would like to know if we can implement any caching or are there any params to optimize the upload.

[ChrisW67]

No, you need to work out what your program is doing the rest of the time. Does uploading a 100MB file using my code take as long?

[darshan.hardas]

yes, the code snippet also takes the same time... uploads 100MB in 11-12mins.

I have noticed that the file below 50MB gets uploaded faster as compared to large file. We can configure Qt for large file support but 100MB file is not considered as large file.

[ChrisW67]

The code I gave you uploads 104857600 bytes on my LAN in 9.48 seconds average at approx 88 Mbps (i.e. saturating the 100Mbps LAN). The CPU effort barely registers. The result scales to a 1GB file in about 100 seconds.

You are claiming about 1.3 Mbps as a Qt imposed limit when other program have no such limit. I don't think the problem is in Qt networking.

[wysota]

Which version of Qt are both of you guys using?

[ChrisW67]

Qt 4.8.2 and Qt 5.0.0Beta1 on Linux talking to a pure-ftpd server on Linux.

I just tried Qt 4.8.0 and Qt 4.7.4 on Windows XP (in a VM on Linux, same server) with the result of about 2100 ms for the 10MB file, 20500ms for the 100MB file. FileZilla on the same machine manages to match Linux; a shade under 10 seconds for the 100MB file. On Windows 7 the performance is even worse, although I cannot see an obvious cause.

Looking with Wireshark: both transfers from Windows are in FTP passive mode. The data packets are the same size. The pattern of outgoing data an incoming ACK packets is quite different, with Qt bunching outgoing packets more than Filezilla. QTBUG-16359 may be pertinent

[darshan.hardas]

I also have tried with Qt 4.8.2, 4.7.4 on Windows. The transfer speed is very slow. We are also targeting to use the static libs of Qt. But firstly the transfer rate should fair enough to handle atleast 500 MB of file.

[wysota]

I also have tried with Qt 4.8.2, 4.7.4 on Windows. The transfer speed is very slow. We are also targeting to use the static libs of Qt. But firstly the transfer rate should fair enough to handle atleast 500 MB of file.

So have you launched the application that Chris posted as-is or did you somehow changed it or incorporated it in your program?

[ChrisW67]

To be clear, my code example is markedly slower on Windows too. I tried 4.7.4 and 4.8.0 using MingW. I have not tried MSVC to see it it is a C++ runtime issue.

[wysota]

To be clear, my code example is markedly slower on Windows too.

But not as slow as to transfer 100MB in 11-12 minutes.

[ChrisW67]

No, granted. But I am working across a LAN, and the OP is not.

For 100 Mb the time taken is approx 12 mins by QFTP/QNetworkAcessManager. But the ftp clients like FileZilla uploads the file in 3-4 mins.

So, the "optimum" is about 3 times faster than the Qt program (500 vs 150 KB/s). This is not that dissimilar the slowdown factor I saw on a LAN connection with XP. Windows 7 was worse again.

[darshan.hardas]

I have used the code-as is posted. Also, have implemented the logic using QFtp.

We are testing the speed using machine instances created on cloud (Amazon EC2). The network bandwidth is very high on both the instances i.e. client and server.

[darshan.hardas]

Finally I have to try using the sockets QTcpSocket and have to implemented the whole FTP RFC to transfer the file. I have managed to read the file in the separate thread and storing it in buffer to be transfered onto the connection.

This methodology have increased the upload speed and now I can transfer the 100 MB is 3-4 minutes.

Thank you all for giving your inputs.

[Raketten46]

This sounds like a lot of work, can we do something about this?

[cygnus2k]

Hi, i have used your code, and get this error:

[root@mcapro Upload]# ./Upload
QIODevice::read: device not open
Uploaded 0 of 0
Finished 201

The file exists and have size

[ChrisW67]

The error message is clear, the file is not open. If you are using a relative path to open the file then it is likely that the current working directory of the process is not what you think it is.