Question about ftp using Qt
Hi,
I need to ftp 30000 files from a particular website with same username and password, I am using the following code.
My question: will the ftp login 30000 times, or will it keep the connection alive during the entire ftp session until all files are ftped? Also how to I set timeout?
Thanks
Code:
url.setUserName( "ftp" );
url.setPassword( "xxx" );
QNetworkAccessManager manager;
connect( &manager, SIGNAL( finished( QNetworkReply* ) ),
&eventLoop, SLOT( quit() ) );
foreach ( const QString& file, fileList ) {
QUrl.
setUurl( "ftp://ftp.xxx.com/" + file );
QNetworkReply *reply = manager.get( QNetworkRequest( url ) );
/// block until each file is finished, how do I set timeout in the next line?
eventLoop.exec();
/// get the reply content
reply->readAll();
....
reply->deleteLater();
}
Re: Question about ftp using Qt
QNetworkAccessManager will reuse connections for HTTP but I have never investigated FTP. You can check with Wireshark fairly quickly. In your example (clearly not your real code) you are not sending credentials so there might only be an anonymous FTP login.
If you were not explicitly serialising your requests QNetworkAccessManager will also process several at the same time. It would be a good idea to check if the transfer was successful before you rely on the data.
Set the timeout for what?
Re: Question about ftp using Qt
Thanks!
Set timeout for connection or maximum duration of the ftp job. I have managed to use QTimer for that.
However, I now have another problem.
I connect to QNetworkAccessManager::finished signal, and then read data using reply->readAll(). Problem is reply sometimes appends new line "\n" in between data chunk, causing the xml file to break.
For example, tag <conversion> becomes:
<convers
ion>
This cause xml parser to fail. The xml file itself has many new lines, so I don't to use QString::replace( "\n", "" ), because this will cause entire xml file to be a single line.
How do I fix this problem?
Thanks
Re: Question about ftp using Qt
Quote:
I connect to QNetworkAccessManager::finished signal, and then read data using reply->readAll(). Problem is reply sometimes appends new line "\n" in between data chunk, causing the xml file to break.
No, Qt doesn't. That raises the question, what are you actually doing to the QByteArray after readAll() gives it to you?
Re: Question about ftp using Qt
Quote:
Originally Posted by
ChrisW67
No, Qt doesn't. That raises the question, what are you actually doing to the QByteArray after readAll() gives it to you?
I did:
QString content = reply->readAll();
Then parse the content using xml parser, and found that "\n" is inserted in between data in some of the files.
Re: Question about ftp using Qt
Is the FTP server public? Example file?
Re: Question about ftp using Qt
Quote:
Originally Posted by
lni
QString content = reply->readAll();
That does an uncontrolled 8-bit encoding to 16-bit unicode conversion, usually not something you want to do when processing XML. The XML document usually has a processing instruction for the correct encoding and the XML parsers react to that.
Which XML parser are you using?
Cheers,
_
Re: Question about ftp using Qt
I am now pretty sure QNetworkReply append "\n" in between chunk of data (not all files though).
I switch to QByteArray, the same thing happens.
It complains:
ERROR: code = 3 , what = "Expected '>', but got '[a-zA-Z]'."
Here is part of the code:
Code:
file.write( content );
// parse the file using QXmlStreamReader
QXmlStreamReader reader( content );
...
I look at the file, some of the xml tag is broken into 2 lines.
There are only English characters in the content. No other language.
And I hit another problem: I submit a job to ftp 300 files, one of the files doesn't exist, it emits error code 203 (ContentNotFoundError). This is fine. But immediately, all remain files got 201 error code (ContentAccessDenied).
If I remove the bad files, the ftp runs without problem. It get the files. As soon as I put back the non-existing file, 203 is emitted, and immediately the rest files got 201, and job stops.
I am using Qt 4.8.1 under CentOS.
PS. I Just google search, and found someone is having the same problem: http://www.qtforum.org/article/33629...essdenied.html
Re: Question about ftp using Qt
Very strange.
You could alternatively try to use QFtp.
Cheers,
_
Re: Question about ftp using Qt
Quote:
Originally Posted by
lni
And I hit another problem: I submit a job to ftp 300 files, one of the files doesn't exist, it emits error code 203 (ContentNotFoundError). This is fine. But immediately, all remain files got 201 error code (ContentAccessDenied).
If I remove the bad files, the ftp runs without problem. It get the files. As soon as I put back the non-existing file, 203 is emitted, and immediately the rest files got 201, and job stops.
That may be the proper behavior of the ftp server you are connecting to.
Re: Question about ftp using Qt
Quote:
I am now pretty sure QNetworkReply append "\n" in between chunk of data (not all files though).
Until you demonstrate that the server is not sending these newline characters I am going to assume you are mistaken. If this was a normal behaviour of QNetworkReply on FTP links this would not bode well for lots of Qt FTP usage and the bug tracker would be running hot.
I just tested fetching a few dozen XML files without a byte out of place (compared byte for byte with version gathered by rsync):
Code:
#include <QtCore>
#include <QtNetwork>
int main(int argc, char **argv)
{
<< "assistant/metadata.xml"
<< "qtsvg/metadata.xml"
<< "qtopenvg/metadata.xml"
<< "qtphonon/metadata.xml"
<< "qt3support/metadata.xml"; // ... plenty more like these
QNetworkAccessManager manager;
QObject::connect(&manager,
SIGNAL(finished
(QNetworkReply
*)),
&eventLoop,
SLOT(quit
()));
foreach (const QString& file, fileList ) {
QUrl url
("ftp://mirror.internode.on.net/pub/gentoo-portage/dev-qt/" + file);
url.setUserName( "anonymous" );
url.setPassword( "anon@example.com" );
QNetworkReply *reply = manager.get( QNetworkRequest( url ) );
/// block until each file is finished, how do I set timeout in the next line?
eventLoop.exec();
/// get the reply content
qDebug() << file << "Error" << reply->error();
QFile out
(outFileName.
replace("/",
"_"));
out.write(data);
out.close();
}
reply->deleteLater();
}
return 0;
}
For the record:
- QNetworkAccessManager only executed a single FTP login (USER/PASS).
- If a file did not exist I got the same behaviour as you did on subsequent files. It seems that QNetworkAccessManager closed the TCP connection on the FTP 550 response and did not try to reopen it for the later ones. This might be a bug.
Re: Question about ftp using Qt
The file size I try to ftp is around 10,000 byte counts, if this information is helpful. Your sample xml file size perhaps is too small (my random thought).
I didn't block each file, instead I submit all ftp jobs, and then wait until all finish or timeout.
However, it is more serious for 201 error caused by 203 error. I switch to Qt 5.2, same problem exists.
Code:
#include <QtCore>
#include <QtNetwork>
#include <QDebug>
int main(int argc, char **argv)
{
QNetworkAccessManager manager;
QObject::connect(&manager,
SIGNAL(finished
(QNetworkReply
*)),
&eventLoop,
SLOT(quit
()));
QList<QNetworkReply*> replies;
int loops = 200;
for ( int idx = 0; idx < loops; idx++ ) {
/// let's ftp same file 200 times and compare them
/// (But I just got first file, why?)
QUrl url
( "ftp://mirror.internode.on.net/pub/gentoo-portage/dev-qt/assistant/metadata.xml" );
url.setUserName( "anonymous" );
url.setPassword( "anon@example.com" );
replies << manager.get( QNetworkRequest( url ) );
}
/// you need to set time out before exec is call
/// allow 60 seconds timeout for each ftp
QTimer::singleShot( loops
* 60 * 1000,
&eventLoop,
SLOT( quit
() ) );
/// block until all files area finished
eventLoop.exec();
qDebug() << "done";
qDebug() << ba0;
for ( int idx = 1; idx < loops; idx++ ) {
if ( ba0.size() != ba.size() ) {
qDebug() << idx << ": Size not equal";
qDebug() << ba;
break;
}
}
return 0;
}