Ftp PUT gets stuck on Windows

[vojta]

Hi,

I have written application that uploads files from directory to FTP on my server. It works fine on Linux, but when I try it on Windows it gets stuck almost always. It just stops uploading files (no errors). I have already tried using QFtp and QNetworkAccessManager.

Do you have any idea what could be wrong, please?

Thanks in advance,
vojta

[wysota]

You could be filling some buffer somewhere in the system.

[vojta]

Hmm, yea. However, I have no idea what buffer it could be. Each file is ~400 KB PNG image. I always open QFile and put it using QFtp or QNetworkAccessManager. When put gets finished I close QFile and then do this again with another file. I have tried running app with GDB but it does not give me any meaningful information. I guess I just have to try to Google more and more. Anyway, thanks. And if you would have any other idea, please let me know.

[wysota]

Yeah... or you could show us some code.

[vojta]

Well, here is my code. Thanks.

PresentationUploader.h:

#ifndef PRESENTATIONUPLOADER_H
#define PRESENTATIONUPLOADER_H
 
#include <QObject>
#include <QStringList>
 
class QFtp;
 
class PresentationUploader : public QObject
{
        Q_OBJECT
 
    public:
        enum Result
        {
            Nothhing,
            Success = 1,
            AlreadyRunning,
            BadInput,
            PresentationDirectoryDoesNotExist,
            EmptyPresentationDirectory,
            AbortedByUser,
            CannotConnectToFtpServer,
            FtpServerLoginFailed,
            ChangeDirectoryFailed,
            MakeDirectoryFailed,
            CannotOpenFile,
            FileUploadFailed
        };
 
        PresentationUploader();
 
        void setPresentationId(int presentationId);
        void setDirectory(const QString& directory);
        void setFtpHost(const QString& host);
        void setFtpDirectory(const QString& directory);
        void setFtpCredentials(const QString& username, const QString& password);
 
    public slots:
        void startUpload();
        void abortUpload(PresentationUploader::Result result = PresentationUploader::AbortedByUser);
 
    private slots:
        void ftpProgress(qint64 doneBytes, qint64 totalBytes);
        void ftpCommandFinished(int commandId, bool error);
 
    private:
        void makePresentationDirectory();
        void putNextFile();
 
    private:
        int m_presentationId;
        QString m_directory;
        QString m_ftpHost;
        QString m_ftpDirectory;
        QString m_ftpUsername, m_ftpPassword;
        QFtp* m_ftp;
 
        qint64 m_doneBytes;
        qint64 m_totalBytes;
        QStringList m_files;
 
        PresentationUploader::Result m_result;
        bool m_aborted;
 
    signals:
        void nextFile(const QString& file);
        void totalProgress(qint64 doneBytes, qint64 totalBytes);
        void transferProgress(qint64 doneBytes, qint64 totalBytes);
        void finished(int result);
};
 
#endif // PRESENTATIONUPLOADER_H

#ifndef PRESENTATIONUPLOADER_H
#define PRESENTATIONUPLOADER_H

#include <QObject>
#include <QStringList>

class QFtp;

class PresentationUploader : public QObject
{
        Q_OBJECT

    public:
        enum Result
        {
            Nothhing,
            Success = 1,
            AlreadyRunning,
            BadInput,
            PresentationDirectoryDoesNotExist,
            EmptyPresentationDirectory,
            AbortedByUser,
            CannotConnectToFtpServer,
            FtpServerLoginFailed,
            ChangeDirectoryFailed,
            MakeDirectoryFailed,
            CannotOpenFile,
            FileUploadFailed
        };

        PresentationUploader();

        void setPresentationId(int presentationId);
        void setDirectory(const QString& directory);
        void setFtpHost(const QString& host);
        void setFtpDirectory(const QString& directory);
        void setFtpCredentials(const QString& username, const QString& password);

    public slots:
        void startUpload();
        void abortUpload(PresentationUploader::Result result = PresentationUploader::AbortedByUser);

    private slots:
        void ftpProgress(qint64 doneBytes, qint64 totalBytes);
        void ftpCommandFinished(int commandId, bool error);

    private:
        void makePresentationDirectory();
        void putNextFile();

    private:
        int m_presentationId;
        QString m_directory;
        QString m_ftpHost;
        QString m_ftpDirectory;
        QString m_ftpUsername, m_ftpPassword;
        QFtp* m_ftp;

        qint64 m_doneBytes;
        qint64 m_totalBytes;
        QStringList m_files;

        PresentationUploader::Result m_result;
        bool m_aborted;

    signals:
        void nextFile(const QString& file);
        void totalProgress(qint64 doneBytes, qint64 totalBytes);
        void transferProgress(qint64 doneBytes, qint64 totalBytes);
        void finished(int result);
};

#endif // PRESENTATIONUPLOADER_H

To copy to clipboard, switch view to plain text mode 

PresentationUploader.cpp:

#include "PresentationUploader.h"
 
#include <QBuffer>
#include <QDebug>
#include <QDir>
#include <QFileInfo>
#include <QFtp>
 
PresentationUploader::PresentationUploader() :
    QObject(),
    m_presentationId(-1),
    m_ftp(0),
    m_result(Nothhing)
{
}
 
void PresentationUploader::setPresentationId(int presentationId)
{
    m_presentationId = presentationId;
}
 
void PresentationUploader::setDirectory(const QString& directory)
{
    m_directory = directory;
}
 
void PresentationUploader::setFtpHost(const QString& host)
{
    m_ftpHost = host;
}
 
void PresentationUploader::setFtpDirectory(const QString& directory)
{
    m_ftpDirectory = directory;
}
 
void PresentationUploader::setFtpCredentials(const QString& username, const QString& password)
{
    m_ftpUsername = username;
    m_ftpPassword = password;
}
 
void PresentationUploader::startUpload()
{
    if (m_ftp != 0)
    {
        emit finished(AlreadyRunning);
        return;
    }
 
    if (m_presentationId < 0 || m_directory.isEmpty() || m_ftpHost.isEmpty() || m_ftpDirectory.isEmpty() || m_ftpUsername.isEmpty() || m_ftpPassword.isEmpty())
    {
        emit finished(BadInput);
        return;
    }
 
    QFileInfo directoryInfo(m_directory);
    if (!directoryInfo.exists())
    {
        emit finished(PresentationDirectoryDoesNotExist);
        return;
    }
 
    m_doneBytes = 0;
    m_totalBytes = 0;
    m_files.clear();
 
    QDir directory(m_directory);
    foreach(const QFileInfo& fileInfo, directory.entryInfoList(QDir::Files))
    {
        m_files.append(fileInfo.absoluteFilePath());
        m_totalBytes += fileInfo.size();
    }
 
    if (m_files.isEmpty())
    {
        emit finished(EmptyPresentationDirectory);
        return;
    }
 
    m_ftp = new QFtp(this);
 
    connect(m_ftp, SIGNAL(commandFinished(int, bool)), this, SLOT(ftpCommandFinished(int, bool)));
    connect(m_ftp, SIGNAL(dataTransferProgress(qint64, qint64)), this, SLOT(ftpProgress(qint64, qint64)));
    connect(m_ftp, SIGNAL(dataTransferProgress(qint64, qint64)), this, SIGNAL(transferProgress(qint64, qint64)));
 
    m_ftp->connectToHost(m_ftpHost, 21);
    m_ftp->login(m_ftpUsername, m_ftpPassword);
    m_ftp->cd(m_ftpDirectory);
}
 
void PresentationUploader::abortUpload(PresentationUploader::Result result)
{
    m_result = result;
    m_ftp->close();
}
 
void PresentationUploader::ftpProgress(qint64 doneBytes, qint64 totalBytes)
{
    Q_UNUSED(totalBytes);
 
    emit totalProgress(m_doneBytes + doneBytes, m_totalBytes);
}
 
void PresentationUploader::ftpCommandFinished(int commandId, bool error)
{
    Q_UNUSED(commandId);
 
    if (m_ftp->currentCommand() == QFtp::ConnectToHost)
    {
        if (error)
            abortUpload(CannotConnectToFtpServer);
    }
    else if (m_ftp->currentCommand() == QFtp::Login)
    {
        if (error)
            abortUpload(FtpServerLoginFailed);
    }
    else if (m_ftp->currentCommand() == QFtp::Cd)
    {
        if (error)
            abortUpload(ChangeDirectoryFailed);
        else
            makePresentationDirectory();
    }
    else if (m_ftp->currentCommand() == QFtp::Mkdir)
    {
        /*if (error)
            abortUpload(MakeDirectoryFailed);
        else*/
        putNextFile();
    }
    else if (m_ftp->currentCommand() == QFtp::Put)
    {
        m_doneBytes += m_ftp->currentDevice()->size();
 
        m_ftp->currentDevice()->close();
        m_ftp->currentDevice()->deleteLater();
 
        if (error)
            abortUpload(FileUploadFailed);
        else if (!m_files.empty())
            putNextFile();
        else if (!m_ftp->hasPendingCommands())
            abortUpload(Success);
    }
    else if (m_ftp->currentCommand() == QFtp::Close)
    {
        m_ftp->deleteLater();
        m_ftp = 0;
 
        emit finished(m_result);
    }
}
 
void PresentationUploader::makePresentationDirectory()
{
    m_ftp->mkdir(QString::number(m_presentationId));
}
 
void PresentationUploader::putNextFile()
{
    QString filePath(m_files.takeFirst());
    QFileInfo fileInfo(filePath);
    QFile* file = new QFile(filePath);
 
    emit nextFile(fileInfo.fileName());
 
    if (!file->open(QIODevice::ReadOnly))
        abortUpload(CannotOpenFile);
 
    m_ftp->put(file, QString("%1/%2").arg(m_presentationId).arg(fileInfo.fileName()));
 
    if (m_files.isEmpty()) // put complete file after uploading last file
    {
        QByteArray* byteArray = new QByteArray("OK");
        QBuffer* buffer = new QBuffer(byteArray);
 
        m_ftp->put(buffer, QString("%1/complete").arg(m_presentationId));
    }
}

#include "PresentationUploader.h"

#include <QBuffer>
#include <QDebug>
#include <QDir>
#include <QFileInfo>
#include <QFtp>

PresentationUploader::PresentationUploader() :
    QObject(),
    m_presentationId(-1),
    m_ftp(0),
    m_result(Nothhing)
{
}

void PresentationUploader::setPresentationId(int presentationId)
{
    m_presentationId = presentationId;
}

void PresentationUploader::setDirectory(const QString& directory)
{
    m_directory = directory;
}

void PresentationUploader::setFtpHost(const QString& host)
{
    m_ftpHost = host;
}

void PresentationUploader::setFtpDirectory(const QString& directory)
{
    m_ftpDirectory = directory;
}

void PresentationUploader::setFtpCredentials(const QString& username, const QString& password)
{
    m_ftpUsername = username;
    m_ftpPassword = password;
}

void PresentationUploader::startUpload()
{
    if (m_ftp != 0)
    {
        emit finished(AlreadyRunning);
        return;
    }

    if (m_presentationId < 0 || m_directory.isEmpty() || m_ftpHost.isEmpty() || m_ftpDirectory.isEmpty() || m_ftpUsername.isEmpty() || m_ftpPassword.isEmpty())
    {
        emit finished(BadInput);
        return;
    }

    QFileInfo directoryInfo(m_directory);
    if (!directoryInfo.exists())
    {
        emit finished(PresentationDirectoryDoesNotExist);
        return;
    }

    m_doneBytes = 0;
    m_totalBytes = 0;
    m_files.clear();

    QDir directory(m_directory);
    foreach(const QFileInfo& fileInfo, directory.entryInfoList(QDir::Files))
    {
        m_files.append(fileInfo.absoluteFilePath());
        m_totalBytes += fileInfo.size();
    }

    if (m_files.isEmpty())
    {
        emit finished(EmptyPresentationDirectory);
        return;
    }

    m_ftp = new QFtp(this);

    connect(m_ftp, SIGNAL(commandFinished(int, bool)), this, SLOT(ftpCommandFinished(int, bool)));
    connect(m_ftp, SIGNAL(dataTransferProgress(qint64, qint64)), this, SLOT(ftpProgress(qint64, qint64)));
    connect(m_ftp, SIGNAL(dataTransferProgress(qint64, qint64)), this, SIGNAL(transferProgress(qint64, qint64)));

    m_ftp->connectToHost(m_ftpHost, 21);
    m_ftp->login(m_ftpUsername, m_ftpPassword);
    m_ftp->cd(m_ftpDirectory);
}

void PresentationUploader::abortUpload(PresentationUploader::Result result)
{
    m_result = result;
    m_ftp->close();
}

void PresentationUploader::ftpProgress(qint64 doneBytes, qint64 totalBytes)
{
    Q_UNUSED(totalBytes);

    emit totalProgress(m_doneBytes + doneBytes, m_totalBytes);
}

void PresentationUploader::ftpCommandFinished(int commandId, bool error)
{
    Q_UNUSED(commandId);

    if (m_ftp->currentCommand() == QFtp::ConnectToHost)
    {
        if (error)
            abortUpload(CannotConnectToFtpServer);
    }
    else if (m_ftp->currentCommand() == QFtp::Login)
    {
        if (error)
            abortUpload(FtpServerLoginFailed);
    }
    else if (m_ftp->currentCommand() == QFtp::Cd)
    {
        if (error)
            abortUpload(ChangeDirectoryFailed);
        else
            makePresentationDirectory();
    }
    else if (m_ftp->currentCommand() == QFtp::Mkdir)
    {
        /*if (error)
            abortUpload(MakeDirectoryFailed);
        else*/
        putNextFile();
    }
    else if (m_ftp->currentCommand() == QFtp::Put)
    {
        m_doneBytes += m_ftp->currentDevice()->size();

        m_ftp->currentDevice()->close();
        m_ftp->currentDevice()->deleteLater();

        if (error)
            abortUpload(FileUploadFailed);
        else if (!m_files.empty())
            putNextFile();
        else if (!m_ftp->hasPendingCommands())
            abortUpload(Success);
    }
    else if (m_ftp->currentCommand() == QFtp::Close)
    {
        m_ftp->deleteLater();
        m_ftp = 0;

        emit finished(m_result);
    }
}

void PresentationUploader::makePresentationDirectory()
{
    m_ftp->mkdir(QString::number(m_presentationId));
}

void PresentationUploader::putNextFile()
{
    QString filePath(m_files.takeFirst());
    QFileInfo fileInfo(filePath);
    QFile* file = new QFile(filePath);

    emit nextFile(fileInfo.fileName());

    if (!file->open(QIODevice::ReadOnly))
        abortUpload(CannotOpenFile);

    m_ftp->put(file, QString("%1/%2").arg(m_presentationId).arg(fileInfo.fileName()));

    if (m_files.isEmpty()) // put complete file after uploading last file
    {
        QByteArray* byteArray = new QByteArray("OK");
        QBuffer* buffer = new QBuffer(byteArray);

        m_ftp->put(buffer, QString("%1/complete").arg(m_presentationId));
    }
}

To copy to clipboard, switch view to plain text mode 

I move PresentationUploader object to its own thread after creation like this:

QThread* uploaderThread = new QThread();
PresentationUploader* uploader = new PresentationUploader();
 
uploader.moveToThread(uploaderThread);
 
connect(uploaderThread, SIGNAL(started()), uploader, SLOT(startUpload()));
...
uploaderThread->start();

QThread* uploaderThread = new QThread();
PresentationUploader* uploader = new PresentationUploader();

uploader.moveToThread(uploaderThread);

connect(uploaderThread, SIGNAL(started()), uploader, SLOT(startUpload()));
...
uploaderThread->start();

To copy to clipboard, switch view to plain text mode 

[wysota]

What's the thread for? There is no need for that.

[vojta]

Main thread is quite busy because of other functions of application. I wanted to save some resources in main thread and keep it responsive as much as possible. Do you think I can run upload in main thread? I will give it a try.

[wysota]

You can definitely do all that in the main thread. If you want to offload some work to a different thread then offload cpu intensive operations from the main thread.

[vojta]

Ok, I have removed that thread and I am running upload in main thread, now. However, it still gets stuck sometimes.

[wysota]

Add debug messages to your methods (or use a debugger) to see the path the code is taking. Maybe the upload gets aborted or something. Also output volume of data that has already been sent. If the volume will be a multiple of 256 then it's probable that some buffer is full.

[vojta]

I have added debug messages and this is the end of output:

FTP: command "Put" id 22 "finished (error: 0 No error)" 
FTP: command "Put" id 23 "queued 117956 bytes 38888923/0-0019.png" 
FTP: command "Put" id 23 "started" 
FTP: progress "Put" id 23 0 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 16384 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 32768 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 49152 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 65536 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 81920 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 98304 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 114688 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 117956 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: command "Put" id 23 "finished (error: 0 No error)" 
FTP: command "Put" id 24 "queued 117956 bytes 38888923/0-0020.png" 
FTP: command "Put" id 24 "started" 
FTP: progress "Put" id 24 0 b of 117956 b 
     currentDevice "E:/data/0-0020.png" error: "No error"

FTP: command "Put" id 22 "finished (error: 0 No error)" 
FTP: command "Put" id 23 "queued 117956 bytes 38888923/0-0019.png" 
FTP: command "Put" id 23 "started" 
FTP: progress "Put" id 23 0 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 16384 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 32768 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 49152 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 65536 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 81920 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 98304 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 114688 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: progress "Put" id 23 117956 b of 117956 b 
     currentDevice "E:/data/0-0019.png" error: "No error" 
FTP: command "Put" id 23 "finished (error: 0 No error)" 
FTP: command "Put" id 24 "queued 117956 bytes 38888923/0-0020.png" 
FTP: command "Put" id 24 "started" 
FTP: progress "Put" id 24 0 b of 117956 b 
     currentDevice "E:/data/0-0020.png" error: "No error"

To copy to clipboard, switch view to plain text mode 

And here application gets stuck. It always gets stuck at the beginning of file. It looks like QFtp started Put command but does not upload anything and it really does not send any data over network (checked with Wireshark).

[wysota]

Is it possible for you to try that with a different server?

[ChrisW67]

Not the cause of the current problem, unless memory is being exhausted, but the memory leaks in PresentationUploader:utNextFile() are obvious. Using the heap here is unnecessary. The same is probably true of the m_ftp object in the face of calling abortUpdate().

[Spitfire]

Maybe you need to set passive or active mode for the ftp connection?

On my work windows box I can't use anything else than passive ftp connection regardles of application used.
If I use active connection the upload will just sit there doing nothing like yours so maybe it's the same reason.

[vojta]

Is it possible for you to try that with a different server?

I have just tried different server and same problem occured. However, both were vsftpd. I will try it with proftpd as soon as possible.

Not the cause of the current problem, unless memory is being exhausted, but the memory leaks in PresentationUploader:utNextFile() are obvious. Using the heap here is unnecessary. The same is probably true of the m_ftp object in the face of calling abortUpdate().

I have already fixed those memleaks, but thanks.

Maybe you need to set passive or active mode for the ftp connection?

On my work windows box I can't use anything else than passive ftp connection regardles of application used.
If I use active connection the upload will just sit there doing nothing like yours so maybe it's the same reason.

QFtp::Passive is default transfer mode for QFtp. However, I have tried to set it explicitely and same problem occured. :/

[Spitfire]

Passive works for me, but it doesn't have to for you.
Just to be sure try active mode, maybe it's as simple as that.
If not - then good luck

[vojta]

I can't use Active, because application must work behind NAT.