Results 1 to 7 of 7

Thread: [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with proxy

  1. #1
    Join Date
    Jun 2015
    Posts
    4
    Qt products
    Qt5
    Platforms
    Windows

    Default [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with proxy

    I am using qt(5.3) on windows 7(vs2008)(below code, skipping the details) to upload a file which is on the disk. if the proxy is not enabled on the system,
    upload is working fine for any size(i tried ~3gb) without spiking memory for my application(~22MB). But as i enable system proxy, memory for the application
    is growing propotional to uploadProgress. It works fine with file size below 1.3GB upload. As my application is 32 bit, it crashes(bad_alloc, in qt) if i try to
    upload more than 1.3GB file with proxy.

    So i have following queries:
    1) Are we loading complete file into memory if proxy is enabled?

    2) As i guess, we are not loading complete file into memory if proxy is not enabled. Can we do same thing if proxy is enabled? If yes, how can we achieve that?




    Qt Code:
    1. QNetworkReply* upload(const QString &title, const QString &path, const QString &payload, QNetworkRequest request, QNetworkAccessManager* nam)
    2.  
    3.  
    4.  
    5. QHttpMultiPart *pMultipart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
    6. QHttpPart richPart;
    7. richPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"entity_document\""));
    8. richPart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("application/json; charset=UTF-8"));
    9. richPart.setBody(payload.toUtf8());
    10.  
    11. QString binaryHeader = "form-data; name=\"fileData\"; filename=\"";
    12. binaryHeader += title;
    13. binaryHeader += "\"";
    14.  
    15. QHttpPart binaryHeader;
    16. binaryHeader.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("text/plain; charset=UTF-8"));
    17. binaryHeader.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant(binaryHeader));
    18.  
    19. QFile *fileP = new QFile(path);
    20. if(!fileP->open(QIODevice::ReadOnly))
    21. {
    22. //logging the error
    23. }
    24. binaryHeader.setBodyDevice(fileP);
    25. fileP->setParent(pMultipart);
    26.  
    27. pMultipart->append(richPart);
    28. pMultipart->append(binaryHeader);
    29.  
    30. m_replyP = nam->post(request, pMultipart);
    31. pMultipart->setParent(m_replyP);
    32. //bindall(); here binding finished() and uploadProgress signal
    33. return m_replyP;
    34. }
    35.  
    36. //Skipped normal memory cleanup
    To copy to clipboard, switch view to plain text mode 


    Applicatin initialization if Proxy is enabled on machine:
    Qt Code:
    1. QNetworkProxy applicationProxy = QNetworkProxy::applicationProxy();
    2.  
    3. applicationProxy.setUser(user);
    4. applicationProxy.setPassword(password);
    5.  
    6. QNetworkProxy::setApplicationProxy(applicationProxy);
    7. nam->setProxy(applicationProxy);
    To copy to clipboard, switch view to plain text mode 
    Last edited by anda_skoa; 24th June 2015 at 15:39. Reason: missing [code] tags

  2. #2
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with pr

    Connect something to the QNetworkAccessManager::proxyAuthenticationRequired () signal and see if the proxy is rejecting your proxy user/password and expecting new details.

    Once you have set the application proxy you should not need to also set it on the QNetworkAccessManager.

  3. #3
    Join Date
    Jun 2015
    Posts
    4
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with pr

    Thanks for your reply ChrisW67. I will try that and let you know. But as there is no issue with proxy authentication as we are getting that proxy credential dialog box and after giving the credential we are able to upload files with above setups. But as i take any file above 1.3 GB, application crashes with out of memory error. If i take any file less than 1.3GB, it is able to upload through proxy.

    So, what i guess that Qt reads the complete file into buffer/cache before uploading if i am using proxy with the above setup. But if I am not using a proxy then it is not loading the complete file into memory and application size hovers around 22 MB even though if i upload 3GB file. So, is qt using some internal buffer/cache to store that content of file into memory when proxy is enabled? Please let me know if i am not clear.

  4. #4
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with pr

    Is your code snippet your actual code? Lines 11 and 15 are at odds with each other over the type of binaryHeader.

    I cannot see an immediate reason Qt might do this unless the proxy has some odd limitation.

    What does the memory do while you are moving your 1.3GB payload?
    Have you used Fiddler/Wireshark to see if data is actually moving on the net when the proxy is in play?
    What type of proxy are you using? HTTP or SOCKS?
    Is it HTTP or an HTTPS site? The proxying mechanism may be different.

  5. #5
    Join Date
    Jun 2015
    Posts
    4
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with pr

    Thanks for your reply ChrisW67. Please see inline.

    >Is your code snippet your actual code? Lines 11 and 15 are at odds with each other over the type of binaryHeader.

    Both are different, typo while editing the post.


    >> I cannot see an immediate reason Qt might do this unless the proxy has some odd limitation.

    I looked at this thread: http://www.qtcentre.org/threads/3713...ailable-memory

    But for me, large file upload is working for more than 2GB if proxy is not enabled. When proxy is enabled, nam->post executes slighly different qt code as i walk through code. Qt has different code added with #ifProxysomething macro many places. Maybe, the problem is not with normal upload code path if proxy is not enabled.


    >>What does the memory do while you are moving your 1.3GB payload?

    It is spiking up with upload. As when uploadprogress says 10 MB uploaded, same time memory spike in task manager to ~22(normal application size) + 10MB
    for 500MB, memory spikes to ~(22 + 500) MB

    for 1.3GB .................. ~22 + 1.3GB

    >>Have you used Fiddler/Wireshark to see if data is actually moving on the net when the proxy is in play?

    Will check with these and post the findings.

    >> What type of proxy are you using? HTTP or SOCKS?

    We are using HttpProxy(type: 3)
    >> Is it HTTP or an HTTPS site? The proxying mechanism may be different.
    it is https:// site.

    And one more thing i noticed is it is happening if https site is used.
    Last edited by mavericks; 26th June 2015 at 14:44.

  6. #6
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,734
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with pr

    The way HTTPS is proxied your program should issue a CONNECT command to the proxy server and thereafter your connection it is an encrypted transparent tunnel through the proxy server. The tunnel is essentially the same as a standard TCP connection. I cannot see why Qt would buffer the file differently, but that does not mean it is not happening though.

  7. #7
    Join Date
    Jun 2015
    Posts
    4
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [QT5.3:Windows7]File upload through QNetworkAccessManager, memory spiking with pr

    ChrisW67,
    Thanks for your reply. I was going through issues related to https and ssl etc and found this:
    http://www.qtcentre.org/threads/2452...kReply-and-SSL

    I think i am hitting same issue.

    Is there any bug or issue similar to above in qt 5.3.1.

    Above issue is easily reproducible in qt 5.3.1 with sample application, enabled proxy and 'https' upload site.

Similar Threads

  1. FTP Upload using QNetworkAccessManager
    By replax in forum Newbie
    Replies: 2
    Last Post: 30th October 2014, 12:32
  2. Replies: 6
    Last Post: 4th June 2011, 00:07
  3. Replies: 1
    Last Post: 21st October 2010, 04:59
  4. Replies: 5
    Last Post: 20th January 2009, 14:11
  5. upload files using QNetworkAccessManager
    By Raajesh in forum Qt Programming
    Replies: 1
    Last Post: 30th June 2008, 19:43

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.