Multiple QNetworkRequests (post) with QNetworkNetworkAccessManager in a loop

[TimWoo]

Hello all,

This is my first post on here. QtCentre and forum are a great source of information.

I have read through the forum and I found somewhat related posts, but not quite on the money, so I thought I'd give this a go, hoping I haven't missed any existing information on the subject.
As I state in the description, I’m having a problem with sending multiple network requests via the QnetworkAccessManager(QNAM) post method. The windows application runs as a QApplication.

The entire application consists of a client and a server, communicating through Http requests made by the client. The server only responds to requests and doesn't take any 'initiative'.

Sync::Sync(ConnectionInfo info) //info contains ip address and user auth
{
    p = new SyncPrivate;
    p->mManager = new QNetworkAccessManager(this);
    connect(p->mManager, SIGNAL(authenticationRequired(QNetworkReply*,QAuthenticator*)),
            SLOT(authenticate(QNetworkReply*, QAuthenticator*)));
    p->mIpAddress = info.remoteIP();
    switch(info.userLevel()) {
		//UserAuth info here
    }
    p->mPassword = info.password();
}
 
void ControllerListEntry::on_actionStop_Monitoring_All_Nodes_triggered()
{
    if (mSync != 0)
        return;
 
    mSync = new Sync(mController->connectionInfo());
    connect(mSync, SIGNAL(finished()), this, SLOT(stopMonitoringAllNodesFinished()));
    foreach(Node *mNode, mController->getNodes()){ //Looping through all nodes
        if(!mSync->setMonitoring(mNode, false))
            break;
    }
}
 
 
void ControllerListEntry::on_actionStart_Monitoring_All_Nodes_triggered()
{
    if (mSync != 0)
        return;
 
    mSync = new Sync(mController->connectionInfo());
    connect(mSync, SIGNAL(finished()), this, SLOT(startMonitoringAllNodesFinished()));
    foreach(Node *mNode, mController->getNodes()){ //Looping through all nodes
        if(!mSync->setMonitoring(mNode, true))
            break;
    }
}
 
 
//For each iteration of the previous loop, the following method is called:
bool Sync::setMonitoring(Node n, bool enabled)
{
    QNetworkRequest req(QUrl("http://" + p->mIpAddress + "/rpc/set-monitoring"));
    req.setAttribute(QNetworkRequest::HttpPipeliningAllowedAttribute, QVariant(true));
    QByteArray postReq = "nodename=" + encode(n->getName());
    postReq += "&monitoring=" + QByteArray(enabled ? "true" : "false");
    p->mBusy = true;
    p->mReply = p->mManager->post(req, postReq);
 
    connect(p->mReply, SIGNAL(finished()), SLOT(replyFinished()));
 
    return true;
}
 
//For completeness sake
void Sync::replyFinished()
{
    p->mReply->deleteLater();
    p->mBusy = false;
 
    // Check for errors.
    if (p->mReply->error() != QNetworkReply::NoError) {
          emit error(p->mReply->errorString()); //will call slot showing error message
        return;
    }
 
    emit finished(); //ControllerListEntry will do some ui stuff
}

Sync::Sync(ConnectionInfo info) //info contains ip address and user auth
{
    p = new SyncPrivate;
    p->mManager = new QNetworkAccessManager(this);
    connect(p->mManager, SIGNAL(authenticationRequired(QNetworkReply*,QAuthenticator*)),
            SLOT(authenticate(QNetworkReply*, QAuthenticator*)));
    p->mIpAddress = info.remoteIP();
    switch(info.userLevel()) {
		//UserAuth info here
    }
    p->mPassword = info.password();
}

void ControllerListEntry::on_actionStop_Monitoring_All_Nodes_triggered()
{
    if (mSync != 0)
        return;

    mSync = new Sync(mController->connectionInfo());
    connect(mSync, SIGNAL(finished()), this, SLOT(stopMonitoringAllNodesFinished()));
    foreach(Node *mNode, mController->getNodes()){ //Looping through all nodes
        if(!mSync->setMonitoring(mNode, false))
            break;
    }
}


void ControllerListEntry::on_actionStart_Monitoring_All_Nodes_triggered()
{
    if (mSync != 0)
        return;

    mSync = new Sync(mController->connectionInfo());
    connect(mSync, SIGNAL(finished()), this, SLOT(startMonitoringAllNodesFinished()));
    foreach(Node *mNode, mController->getNodes()){ //Looping through all nodes
        if(!mSync->setMonitoring(mNode, true))
            break;
    }
}


//For each iteration of the previous loop, the following method is called:
bool Sync::setMonitoring(Node n, bool enabled)
{
    QNetworkRequest req(QUrl("http://" + p->mIpAddress + "/rpc/set-monitoring"));
    req.setAttribute(QNetworkRequest::HttpPipeliningAllowedAttribute, QVariant(true));
    QByteArray postReq = "nodename=" + encode(n->getName());
    postReq += "&monitoring=" + QByteArray(enabled ? "true" : "false");
    p->mBusy = true;
    p->mReply = p->mManager->post(req, postReq);

    connect(p->mReply, SIGNAL(finished()), SLOT(replyFinished()));

    return true;
}

//For completeness sake
void Sync::replyFinished()
{
    p->mReply->deleteLater();
    p->mBusy = false;

    // Check for errors.
    if (p->mReply->error() != QNetworkReply::NoError) {
          emit error(p->mReply->errorString()); //will call slot showing error message
        return;
    }

    emit finished(); //ControllerListEntry will do some ui stuff
}

To copy to clipboard, switch view to plain text mode 

The problem is that requests are sent for only part of the Nodes in the Controller when above a certain number of Nodes. This certain number is somewhere around 8, this sound a bit vague and it is, but that's the nature of this problem. Sometimes 6 requests are sent, sometimes up to 11. Always more than 6 though.

I'm aware of the following line from the Qt Documentation for QNetworkAccessManager:
Note: QNetworkAccessManagerqueues the requests it receives. The number of requests executed in parallel is dependent on the protocol. Currently, for the HTTP protocol on desktop platforms, 6 requests are executed in parallel for one host/port combination.

That makes me expect that all the requests will be sent in blocks of 6, until no requests are left. Am I wrong in this?
Allowing HTTP Pipelining or disabling it doesn't seem to make any difference in the result.

Sometimes I get the following in the application output: QCoreApplication::postEvent: Unexpected null receiver. Would this happen when not receiving a network reply?

I can confirm:
• The setMonitoring code finished every time for every Node.
• Only for part of the (1 - very small amount) Nodes a reply is caught by the replyFinished Slot
â—‹ Using Wireshark:
§ Http Replies are actually seen on the network
§ Not always an Http reply for every request
• It is always the requests for the last n Nodes that don't get sent. No "gaps" in between.
• Qt version: Qt 4.7.4.

For 18 Nodes this is what happens on the network on Http level between client and server.
WiresharkSetMon.jpg

I hope someone can provide some ideas on the matter.

Many thanks in advance,


Tim

[TimWoo]

Hi all,

I see I could have clarified the structure of the method calls.

The following happens:
* The user clicks a button in the UI, which connects to the on_actionStart_Monitoring_All_Nodes_triggered() and it's "Stop" counterpart.
* Either of these methods loops through all the Node elements in the system and calls the Sync::setMonitoring() method for each of these.
* In the setMonitoring() method, the request is built and posted to the server.

For clarification on how the QNetworkAccessManager is initiated I added the constructor for Sync. I also added the slot that the finished() signal from the QNetworkReply connects to, for completeness sake.

I hope this makes it a bit easier to read.

It's sometimes hard to write up these parts out of their context. For me at least .

Looking forward to your thoughts.

Tim

[yeye_olive]

I do not understand all the details of your code, but something caught my attention. The line

p->mReply = p->mManager->post(req, postReq);

p->mReply = p->mManager->post(req, postReq);

To copy to clipboard, switch view to plain text mode 

overwrites the SyncPrivate's mReply member every time you send a request. Since all the requests made in the loop seem to go through the same Sync instance, I suppose that after the loop, mReply points to the last reply. Then, in Sync::replyFinished(), you always deal with that last reply, even if the slot is probably called for several of them. Here is a suggestion: instead of connecting to the QNetworkReply's finished() signal, consider connecting to QNetworkAccessManager::finished(QNetworkReply *) instead; that way, you get a pointer to the reply in the slot, right where you need it.

[ChrisW67]

The problem is that requests are sent for only part of the Nodes in the Controller when above a certain number of Nodes. This certain number is somewhere around 8, this sound a bit vague and it is, but that's the nature of this problem. Sometimes 6 requests are sent, sometimes up to 11. Always more than 6 though.

As part of being a well-behaved netizen QNetworkAccessManager limits the number of simultaneous connections to a single server address. I am fairly sure this limit is 6 connections in the Qt5 source. Are all your connections to the same host?

[TimWoo]

As part of being a well-behaved netizen QNetworkAccessManager limits the number of simultaneous connections to a single server address. I am fairly sure this limit is 6 connections in the Qt5 source. Are all your connections to the same host?

Thanks for responding.

Only 1 host is concerned in this case. Is a new connection really created for every request in such a "batch" of 6? I don't think it explains why sometimes more than 6 requests (up to 11 observed so far) can be monitored on the network (and always the first n, no gaps in the sequence), do you have an explanation for this?

Please note that my code is written in Qt 4.7. Thought I'd mention it since you talk about Qt5.

I do not understand all the details of your code, but something caught my attention. The line

p->mReply = p->mManager->post(req, postReq);

p->mReply = p->mManager->post(req, postReq);

To copy to clipboard, switch view to plain text mode 

overwrites the SyncPrivate's mReply member every time you send a request. Since all the requests made in the loop seem to go through the same Sync instance, I suppose that after the loop, mReply points to the last reply. Then, in Sync::replyFinished(), you always deal with that last reply, even if the slot is probably called for several of them. Here is a suggestion: instead of connecting to the QNetworkReply's finished() signal, consider connecting to QNetworkAccessManager::finished(QNetworkReply *) instead; that way, you get a pointer to the reply in the slot, right where you need it.

Good point I guess. That would explain why the application only reports replies for some of the requests. Do you think this would influence the behaviour of sending the requests?

[yeye_olive]

Good point I guess. That would explain why the application only reports replies for some of the requests. Do you think this would influence the behaviour of sending the requests?

I do not know the implementation details of QNetworkAccessManager, but it is entirely possible that the manager stops sending requests while too many QNetworkReply are around, waiting for the user to read their contents and then delete them.

[TimWoo]

@ChrisW67
Do you have any further insights?

@yeye_olive
I'm not familiar with the QNetworkAccessManager implementation myself, maybe someone else on the forum is? If so, I would very much welcome the input .

Further:
I cannot find any information about the maximum number of requests offered to QNetworkAccessManager simultaneously in the Qt 4.8 Documentation or anywhere else for that matter. I'm under the impression that it shouldn't be a problem for QNetworkAccessManager, but I could very well be wrong offcourse. It's all quite complex to me, as I'm still not quite familiar with asynchronous message handling.
I'm currently tempted to prototype a workaround to see if offering 6 requests to the QNAM at the same time solves the problem. I would then wait sending the next 6 requests once the previous 6 have been processed.

If anyone can shed some more light on the subject I would be very grateful.

Best Regards,

Tim