Results 1 to 5 of 5

Thread: QNetworkAccessManager and multiple simultaneous requests

  1. #1
    Join Date
    Jul 2015
    Posts
    2
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows Android

    Default QNetworkAccessManager and multiple simultaneous requests

    I need to connect to multiple sites (variable number of sites, may be over 100) and fetch some data in a loop for each site.

    I create a thread and a QNAM for each site. (For each site QNAM must have different cookies).

    All good for few sites, but when sites number is big I get

    "Too many open pipes"

    and crash.

    Then I've tried using a single thread but is too slow because it won't fetch data from next site until current site reply is finished.

    There is some way to limit the number of simultaneous connection to 15 - 20 while the others waiting, and when there is a free slot open connection, do stuff and close connection so that the other threads can do their job ?

  2. #2
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: QNetworkAccessManager and multiple simultaneous requests

    Quote Originally Posted by Angelo Cruz View Post
    I create a thread and a QNAM for each site. (For each site QNAM must have different cookies).
    Different cookies as in "the same non-site specific cookie but each site should get a different value"?

    Quote Originally Posted by Angelo Cruz View Post
    Then I've tried using a single thread but is too slow because it won't fetch data from next site until current site reply is finished.
    You mean one thread and still one QNAM per site?
    Even one QNAM can handle multiple connections simultaniously, so maybe you are blocking the processing somehow?

    Cheers,
    _

  3. #3
    Join Date
    Jul 2015
    Posts
    2
    Qt products
    Qt5
    Platforms
    MacOS X Unix/X11 Windows Android

    Default Re: QNetworkAccessManager and multiple simultaneous requests

    Thank you for your reply!

    First I login to those sites and save the cookies.

    When I use one thread I use one QNAM : I set the received cookies from login (otherwise the site won't accept my requests) , I use a QEventLoop to wait until I get the reply, then I set the QNAM with the cookies from the next site and so on!

    What I'm thinking is to use some kind of global lock with 20 items, each site in his own thread, each thread check if one of 20 slots is free if find a free slot open new QNAM do his job then close QNAM and make free that slot. If no slot is free sleep 200 ms until a slot is freed. (or something like that).

  4. #4
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: QNetworkAccessManager and multiple simultaneous requests

    That sounds overly complicated.

    Can't you set all cookies and then just make your requests and leave the distribution of the jobs to QNAM?

    Cheers,
    _

  5. #5
    Join Date
    Dec 2009
    Location
    New Orleans, Louisiana
    Posts
    791
    Thanks
    13
    Thanked 153 Times in 150 Posts
    Qt products
    Qt5
    Platforms
    MacOS X

    Default Re: QNetworkAccessManager and multiple simultaneous requests

    Did you try this first w/o threading and forcing synchronous behavior? What issues did you run into with the asynchronous approach?

Similar Threads

  1. QNetworkAccessManager and multiples requests
    By hassinoss in forum Qt Programming
    Replies: 7
    Last Post: 26th August 2014, 15:43
  2. Multiple requests with QNetworkAccessManager
    By ljuhrich in forum Qt Programming
    Replies: 7
    Last Post: 7th June 2013, 09:34
  3. Multiple QNetworkAccessManager requests.
    By Diath in forum Qt Programming
    Replies: 1
    Last Post: 13th October 2011, 23:04
  4. Replies: 4
    Last Post: 19th January 2011, 14:49
  5. Tracking multiple requests with QNetwork
    By rbp in forum Qt Programming
    Replies: 4
    Last Post: 1st June 2010, 09:04

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.