QHttp file upload doesn't stream?

[haughki]

Hi All,

Using QT 4.5.2 on Windows XP. Code example at bottom of post.

I have a some file upload code which uses QHttp to send a PUT request to a WebDav server. It works fine except for one major problem: it loads the entire file into memory before it uploads the file. This is a problem with large files, a show stopper if the client runs out of memory.

int QHttp::request ( const QHttpRequestHeader & header, QIODevice * data = 0, QIODevice * to = 0 )

int QHttp::request ( const QHttpRequestHeader & header, QIODevice * data = 0, QIODevice * to = 0 )

To copy to clipboard, switch view to plain text mode 

I'm using QHttp::request to start the request. I can't seem to come up with a device or combination of devices which will stream or otherwise buffer the file from the disk.

My working code (which doesn't stream) just passes in the QFile:

_httpId = _qHttp->request(header, sourceFile);

I tried passing a QDataStream (or TextStream), which is not a QIODevice. Tried to come up with a combination of QBuffer, QFile, just to see if something would work. No luck.

I took a look at the QT source. Interestingly, it _is_ buffering the file read in qhttp.cpp, QHttpPrivate::_q_slotBytesWritten(). The memory load looks to me like it's happening in qsslsocket.cpp, QSslSocket::writeData() when the method memcpy's the buffer.

- Can I use QHttp to do what I want? It doesn't seem like it, but maybe I'm missing something.
- Is this a bug (I don't think so, yet)?
- Is it worth trying QNetworkAccessManager, or am I going to run into the same problem?
- If I need to write custom code to support this, could someone outline a basic approach? Will I need to creat multiple requests for this kind of "chunked" transfer?

Code snippet (builds, haven't tried to run. My actual code has a bunch of classes abstracting/wrapping a lot of this.)

QUrl destUrl(QString("http://www.someDAVserver.com/someFolder/someBigFile.dat"));
    QFile* sourceFile = new QFile("D:\\someLocalDir\\someBigFile.dat");
 
    if (!sourceFile->open(QIODevice::ReadOnly)) {
        //throw
    }
 
    QHttp::ConnectionMode mode = destUrl.scheme().toLower() == "https" ? QHttp::ConnectionModeHttps : QHttp::ConnectionModeHttp;
 
    QHttp http;
    http.setHost(destUrl.host(), mode, destUrl.port() == -1 ? 0 : destUrl.port());
 
    QHttpRequestHeader header("PUT", destUrl.toEncoded(QUrl::RemoveScheme|QUrl::RemoveAuthority));
    header.setValue("Host", destUrl.host());
    header.setValue("Accept-Language", QLocale::system().name());
    header.setValue("Content-Length", QString::number(sourceFile->size()));
 
 
    http.request(header, sourceFile);

QUrl destUrl(QString("http://www.someDAVserver.com/someFolder/someBigFile.dat"));
    QFile* sourceFile = new QFile("D:\\someLocalDir\\someBigFile.dat");

    if (!sourceFile->open(QIODevice::ReadOnly)) {
        //throw
    }

    QHttp::ConnectionMode mode = destUrl.scheme().toLower() == "https" ? QHttp::ConnectionModeHttps : QHttp::ConnectionModeHttp;

    QHttp http;
    http.setHost(destUrl.host(), mode, destUrl.port() == -1 ? 0 : destUrl.port());

    QHttpRequestHeader header("PUT", destUrl.toEncoded(QUrl::RemoveScheme|QUrl::RemoveAuthority));
    header.setValue("Host", destUrl.host());
    header.setValue("Accept-Language", QLocale::system().name());
    header.setValue("Content-Length", QString::number(sourceFile->size()));


    http.request(header, sourceFile);

To copy to clipboard, switch view to plain text mode 

Thanks in advance,

Hawkeye Parker

[wysota]

I'd try QNetworkAccessManager. QHttp is obsolete.

[danc81]

The same will happen with QNetworkAccessManager, it seems that the QSslSocket buffers all the data first.

[haughki]

The same will happen with QNetworkAccessManager, it seems that the QSslSocket buffers all the data first.

Yes indeed. Haven't solved that problem yet, but absolutely: it's SSL that's causing the problem.

Also, fwiw, I did try QNetworkAccessManager and had exactly the same problem...

Hawkeye Parker

[danc81]

I've tracked it down to the fact that when QNativeSocketEnginePrivate::nativeWrite() calls WSASend() it returns SOCKET_ERROR and the reason is WSAEWOULDBLOCK, now this appears to be because under normal HTTP, only chunks of 4096 are sent which are fine but using SSL, chunks of 30kB are sent which causes the socket to still be busy on the next attempt. This is find expect that somewhere it seems to report that the bytes were sent and read more from the input device.

That is as far as I've got with it.

[haughki]

I've tracked it down to the fact that when QNativeSocketEnginePrivate::nativeWrite() calls WSASend() it returns SOCKET_ERROR and the reason is WSAEWOULDBLOCK, now this appears to be because under normal HTTP, only chunks of 4096 are sent which are fine but using SSL, chunks of 30kB are sent which causes the socket to still be busy on the next attempt. This is find expect that somewhere it seems to report that the bytes were sent and read more from the input device.

That is as far as I've got with it.

Thanks: good to know; haven't gotten that far. We're going to have to solve this problem one way or another. I'll certainly update this post when I know more. We also have some sense that there should be a way to better configure/build OPENSSL at the client to get around this. Currently still in Google land....

[danc81]

I don't think the problem lies with OpenSSL, I think the problem is wherever the uploadProgress() signal is fired from is incrementing the number of bytes by what should have been sent and not what was actually sent. If this is corrected (in theory a one-liner), it should at worst fire the signal too many times but with the correct information. This is only my theory and I have yet to prove it...

[mgoetz]

Hi,

I have the following patch for Qt 4.5.x: http://pastebin.ca/1619204
It should apply to Qt 4.6 too.
Could you please see if your issues are solved by this?

Markus Goetz

[mgoetz]

If you are not using QHttp but QNetworkAccessManager, there is a 4.5.x patch out too.

[haughki]

Hi,

I have the following patch for Qt 4.5.x: http://pastebin.ca/1619204
It should apply to Qt 4.6 too.
Could you please see if your issues are solved by this?

Markus Goetz

Hi Markus et al.,

We applied the patch on 4.5.2, and in "basic testing" streaming seems to be working fine now over SLL using QHttp.

Thanks _so_ much for this patch, Markus. It's saved us a pretty big chunk of work.

We'll be testing the patch more thoroughly in the coming weeks (as part of our normal app testing), and we'll post here if anything bad comes up.

One note: I'm not expert with patch, but the diff you posted didn't patch smoothly for me. I ended up applying the patch manually. E.g.:

C:\QHttpPatch>patch --dry-run < qhttp_patch.diff patching file qhttp.cpp Hunk #3 FAILED at 2660.
Hunk #4 FAILED at 3119.
2 out of 4 hunks FAILED -- saving rejects to file qhttp.cpp.rej

The diff looks good to me, so I don't know why patch is complaining about hunks #3 and #4. In any case, it was very straightforward to apply the changes manually.

Thanks again,
Hawkeye Parker