[QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with proxy

[mavericks]

I am using qt(5.3) on windows 7(vs2008)(below code, skipping the details) to upload a file which is on the disk. if the proxy is not enabled on the system,
upload is working fine for any size(i tried ~3gb) without spiking memory for my application(~22MB). But as i enable system proxy, memory for the application
is growing propotional to uploadProgress. It works fine with file size below 1.3GB upload. As my application is 32 bit, it crashes(bad_alloc, in qt) if i try to
upload more than 1.3GB file with proxy.

So i have following queries:
1) Are we loading complete file into memory if proxy is enabled?

2) As i guess, we are not loading complete file into memory if proxy is not enabled. Can we do same thing if proxy is enabled? If yes, how can we achieve that?

QNetworkReply* upload(const QString &title, const QString &path, const QString &payload, QNetworkRequest request, QNetworkAccessManager* nam)
 
 
 
			QHttpMultiPart *pMultipart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
            QHttpPart richPart;
            richPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"entity_document\""));
            richPart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("application/json; charset=UTF-8"));
            richPart.setBody(payload.toUtf8());
 
            QString binaryHeader = "form-data; name=\"fileData\"; filename=\"";
            binaryHeader += title;
            binaryHeader += "\"";
 
            QHttpPart binaryHeader;
            binaryHeader.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("text/plain; charset=UTF-8"));
            binaryHeader.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant(binaryHeader));
 
            QFile *fileP = new QFile(path);
            if(!fileP->open(QIODevice::ReadOnly))
            {
               //logging the error
            }
            binaryHeader.setBodyDevice(fileP);
            fileP->setParent(pMultipart); 
 
            pMultipart->append(richPart);
            pMultipart->append(binaryHeader);
 
            m_replyP = nam->post(request, pMultipart);
            pMultipart->setParent(m_replyP);
			  //bindall(); here binding finished() and uploadProgress signal 
			  return m_replyP;
			}
 
           //Skipped normal memory cleanup

QNetworkReply* upload(const QString &title, const QString &path, const QString &payload, QNetworkRequest request, QNetworkAccessManager* nam)
		    
						
			
			QHttpMultiPart *pMultipart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
            QHttpPart richPart;
            richPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"entity_document\""));
            richPart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("application/json; charset=UTF-8"));
            richPart.setBody(payload.toUtf8());

            QString binaryHeader = "form-data; name=\"fileData\"; filename=\"";
            binaryHeader += title;
            binaryHeader += "\"";

            QHttpPart binaryHeader;
            binaryHeader.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("text/plain; charset=UTF-8"));
            binaryHeader.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant(binaryHeader));

            QFile *fileP = new QFile(path);
            if(!fileP->open(QIODevice::ReadOnly))
            {
               //logging the error
            }
            binaryHeader.setBodyDevice(fileP);
            fileP->setParent(pMultipart); 

            pMultipart->append(richPart);
            pMultipart->append(binaryHeader);

            m_replyP = nam->post(request, pMultipart);
            pMultipart->setParent(m_replyP);
			  //bindall(); here binding finished() and uploadProgress signal 
			  return m_replyP;
			}

           //Skipped normal memory cleanup

To copy to clipboard, switch view to plain text mode 

Applicatin initialization if Proxy is enabled on machine:

QNetworkProxy applicationProxy  = QNetworkProxy::applicationProxy();
 
               applicationProxy.setUser(user);
               applicationProxy.setPassword(password);
 
               QNetworkProxy::setApplicationProxy(applicationProxy);
               nam->setProxy(applicationProxy);

QNetworkProxy applicationProxy  = QNetworkProxy::applicationProxy();
			 
               applicationProxy.setUser(user);
               applicationProxy.setPassword(password);

               QNetworkProxy::setApplicationProxy(applicationProxy);
               nam->setProxy(applicationProxy);

To copy to clipboard, switch view to plain text mode 

[ChrisW67]

Connect something to the QNetworkAccessManager::proxyAuthenticationRequired () signal and see if the proxy is rejecting your proxy user/password and expecting new details.

Once you have set the application proxy you should not need to also set it on the QNetworkAccessManager.

[mavericks]

Thanks for your reply ChrisW67. I will try that and let you know. But as there is no issue with proxy authentication as we are getting that proxy credential dialog box and after giving the credential we are able to upload files with above setups. But as i take any file above 1.3 GB, application crashes with out of memory error. If i take any file less than 1.3GB, it is able to upload through proxy.

So, what i guess that Qt reads the complete file into buffer/cache before uploading if i am using proxy with the above setup. But if I am not using a proxy then it is not loading the complete file into memory and application size hovers around 22 MB even though if i upload 3GB file. So, is qt using some internal buffer/cache to store that content of file into memory when proxy is enabled? Please let me know if i am not clear.

[ChrisW67]

Is your code snippet your actual code? Lines 11 and 15 are at odds with each other over the type of binaryHeader.

I cannot see an immediate reason Qt might do this unless the proxy has some odd limitation.

What does the memory do while you are moving your 1.3GB payload?
Have you used Fiddler/Wireshark to see if data is actually moving on the net when the proxy is in play?
What type of proxy are you using? HTTP or SOCKS?
Is it HTTP or an HTTPS site? The proxying mechanism may be different.

[mavericks]

Thanks for your reply ChrisW67. Please see inline.

>Is your code snippet your actual code? Lines 11 and 15 are at odds with each other over the type of binaryHeader.

Both are different, typo while editing the post.


>> I cannot see an immediate reason Qt might do this unless the proxy has some odd limitation.

I looked at this thread: http://www.qtcentre.org/threads/3713...ailable-memory

But for me, large file upload is working for more than 2GB if proxy is not enabled. When proxy is enabled, nam->post executes slighly different qt code as i walk through code. Qt has different code added with #ifProxysomething macro many places. Maybe, the problem is not with normal upload code path if proxy is not enabled.


>>What does the memory do while you are moving your 1.3GB payload?

It is spiking up with upload. As when uploadprogress says 10 MB uploaded, same time memory spike in task manager to ~22(normal application size) + 10MB
for 500MB, memory spikes to ~(22 + 500) MB

for 1.3GB .................. ~22 + 1.3GB

>>Have you used Fiddler/Wireshark to see if data is actually moving on the net when the proxy is in play?

Will check with these and post the findings.

>> What type of proxy are you using? HTTP or SOCKS?

We are using HttpProxy(type: 3)
>> Is it HTTP or an HTTPS site? The proxying mechanism may be different.
it is https:// site.

And one more thing i noticed is it is happening if https site is used.

[ChrisW67]

The way HTTPS is proxied your program should issue a CONNECT command to the proxy server and thereafter your connection it is an encrypted transparent tunnel through the proxy server. The tunnel is essentially the same as a standard TCP connection. I cannot see why Qt would buffer the file differently, but that does not mean it is not happening though.

[mavericks]

ChrisW67,
Thanks for your reply. I was going through issues related to https and ssl etc and found this:
http://www.qtcentre.org/threads/2452...kReply-and-SSL

I think i am hitting same issue.

Is there any bug or issue similar to above in qt 5.3.1.

Above issue is easily reproducible in qt 5.3.1 with sample application, enabled proxy and 'https' upload site.