Reading web content with QNetworkAccessManager and QNetworkReply

[Tomasz]

Hello!

I've got problem with reading content of web pages using QNetworkAccessManager and QNetworkReply. I've got code:

void MainWindow::getXML()
{
    qDebug() << "Getting content..." << endl;
 
    QNetworkRequest request(QUrl("http://www.google.pl"));
    qDebug() << "Network request..." << endl;
 
    NetRepl = NetAccMan.get(request);
    qDebug() << "Network reply..." << endl;
 
    connect(NetRepl, SIGNAL(readyRead()), this, SLOT(parseXML()));
    qDebug() << "Connect..." << endl;
}
 
void MainWindow::parseXML()
{
    qDebug() << "Ready to parse";
 
    QByteArray newData = NetRepl->read(2048);
 
    qDebug() << newData << endl;
[...]
}

void MainWindow::getXML()
{
    qDebug() << "Getting content..." << endl;
    
    QNetworkRequest request(QUrl("http://www.google.pl"));
    qDebug() << "Network request..." << endl;
    
    NetRepl = NetAccMan.get(request);
    qDebug() << "Network reply..." << endl;
    
    connect(NetRepl, SIGNAL(readyRead()), this, SLOT(parseXML()));
    qDebug() << "Connect..." << endl;
}

void MainWindow::parseXML()
{
    qDebug() << "Ready to parse";

    QByteArray newData = NetRepl->read(2048);

    qDebug() << newData << endl;
[...]
}

To copy to clipboard, switch view to plain text mode 

It reads almost every content, but there are some pages that I can't get content (it doesn't go to parseXML()). For example my WWW server (it's simple http server based on small AVR controller) generates simple pages with data that I want to parse:

<RESP>
<FUNC>some_func</FUNC>
<VAL>true</VAL>
<DATA>
   <D1>1</D1>
   <D2>2</D2>
</DATA>
</RESP>

<RESP>
<FUNC>some_func</FUNC>
<VAL>true</VAL>
<DATA>
   <D1>1</D1>
   <D2>2</D2>
</DATA>
</RESP>

To copy to clipboard, switch view to plain text mode 

Only this and nothing more. Do I need some header? How does it work because I'm a little bit confused.

thanks in advance
best regards
Tomasz

[Tomasz]

When I want to read data using my web browser, everything works great. Maybe, there is other way, than using QNetworkAccessManager and QNetworkReply to read somethig from http server?

thanks in advance
best regards
Tomasz

[ChrisW67]

The QIODevice::readyRead() signal indicates that some data is available, not necessarily a lot of data or all the data. Unless you are incrementally parsing the XML you probably want the QNetworkReply::finished() signal.

[Tomasz]

I've tried to use it as You said, and now It goes to parseXML() but when I'm showing the content of a 'newData' buffer - It's empty. Maybe I'm doing something wrong? Any idea?

I can add, that I've got some CGI functions on my AVR, that I use to control microcontroller, and when I trigger them using QNetworkAccessManager and QNetworkReply all of them works, but I can't still read results.

thanks in advance
best regards
Tomasz

[ChrisW67]

If you are getting nothing at all in the QByteArray then look at QNetworkReply::error() for a clue. You could also check that QNetworkReply::isFinished() returns true.

I assume that NetRepl is a class member variable and that only one request can be outstanding at any time. If you allow more than one request the second and subsequently reply pointers will overwrite earlier ones. This situation may result in an unfinished request being accessed in response to another request completing. Use the QNetworkAccessManager::finished() signal, and don't store the QNetworkReply pointer, if you need multiple requests active.

If any of the responses can possibly be larger than 2048 bytes then you would be safer using readAll().

[Tomasz]

If you are getting nothing at all in the QByteArray then look at QNetworkReply::error() for a clue. You could also check that QNetworkReply::isFinished() returns true. [...]

I've done as You said:

void MainWindow::getXML()
{
    ui->textEdit->append("Getting content...");
 
    QNetworkRequest request(QUrl("[my_address]"));
    ui->textEdit->append("Network request...");
 
    NetRepl = NetAccMan.get(request);
    ui->textEdit->append("Network reply...");
 
    connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
 
    ui->textEdit->append("Connect...");
}
 
void MainWindow::parseXML()
{
    ui->textEdit->append("Ready to download and parse...");
    ui->textEdit->append(QString("isFinished:").append(QString::number(NetRepl->isFinished())));
    ui->textEdit->append(QString("error:").append(QString::number(NetRepl->error())));
 
    QByteArray newData = NetRepl->read(2048);
 
    ui->textEdit->append(QString(newData));
 
    NetRepl->deleteLater();
}

void MainWindow::getXML()
{
    ui->textEdit->append("Getting content...");

    QNetworkRequest request(QUrl("[my_address]"));
    ui->textEdit->append("Network request...");

    NetRepl = NetAccMan.get(request);
    ui->textEdit->append("Network reply...");

    connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));

    ui->textEdit->append("Connect...");
}

void MainWindow::parseXML()
{
    ui->textEdit->append("Ready to download and parse...");
    ui->textEdit->append(QString("isFinished:").append(QString::number(NetRepl->isFinished())));
    ui->textEdit->append(QString("error:").append(QString::number(NetRepl->error())));

    QByteArray newData = NetRepl->read(2048);

    ui->textEdit->append(QString(newData));

    NetRepl->deleteLater();
}

To copy to clipboard, switch view to plain text mode 

In my header file I've got:

[...]
private:
    Ui::MainWindow *ui;
 
    QNetworkReply *NetRepl;
    QNetworkAccessManager NetAccMan;

[...]
private:
    Ui::MainWindow *ui;

    QNetworkReply *NetRepl;
    QNetworkAccessManager NetAccMan;

To copy to clipboard, switch view to plain text mode 

I've used finished() signal of QNetworkReply, as You said earlier (connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));). Now It goes to parseXML() and QNetworkReply::error() returns 2 (QNetworkReply::RemoteHostClosedError) and QNetworkReply::isFinished() returns 1. Buffer (newData) is empty. I thik there is no more than one reqest at time, and any of response isn't bigger than 2048 bytes. Any ideas how can I make it work (in browser it works)?

If it's important I can add that those pages that I want to read are CGI functions, which return some values. Address looks like this: http://ip_address/cgi-bin/cgi_functi...?parameter=xxx.

thanks in advance
best regards
Tomasz

[Tomasz]

Maybe I should use some other methods to achieve what I want? Maybe some other classes? My code works fine when I set address that don't exists (correct IP with page that don't exists) then my server responses with "404 Not Found" and I can read It.

thanks in advance
best regards
Tomasz

[ChrisW67]

Is your AVR generating realistic HTTP headers? A Length: atribute? Can you post the result of this:

wget -S http://your.target.url

wget -S http://your.target.url

To copy to clipboard, switch view to plain text mode 

(I assume you are using a Linux box)

[Tomasz]

Is your AVR generating realistic HTTP headers?

You're right. Actually there where no HTTP headers. I haven't even thought about It. I've modified server source code, and now It works just fine! But I've got one more question - can I use the same "QNetworkReply *NetRepl;" and "QNetworkAccessManager NetAccMan;" a couple times? I mean for example in two different functions:

void MainWindow::getXML1()
{
  QNetworkRequest request(QUrl("[my_address_1]"));
  NetRepl = NetAccMan.get(request);
  connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
}
 
void MainWindow::getXML2()
{
  QNetworkRequest request(QUrl("[my_address_2]"));
  NetRepl = NetAccMan.get(request);
  connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
}

void MainWindow::getXML1()
{
  QNetworkRequest request(QUrl("[my_address_1]"));
  NetRepl = NetAccMan.get(request);
  connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
}

void MainWindow::getXML2()
{
  QNetworkRequest request(QUrl("[my_address_2]"));
  NetRepl = NetAccMan.get(request);
  connect(NetRepl, SIGNAL(finished()), this, SLOT(parseXML()));
}

To copy to clipboard, switch view to plain text mode 

And ofcourse in parseXML() I will have:

void MainWindow::parseXML()
{
  [...]
 
  NetRepl->disconnect();
  NetRepl->deleteLater();
}

void MainWindow::parseXML()
{
  [...]

  NetRepl->disconnect();
  NetRepl->deleteLater();
}

To copy to clipboard, switch view to plain text mode 

Would It (putting deleteLater() only once) be enough to get rid of all unused stuff after downloading XML content? Event If one function is triggered after other? I'm asking because class reference says this about deleteLater():

Note that entering and leaving a new event loop (e.g., by opening a modal dialog) will not perform the deferred deletion; for the object to be deleted, the control must return to the event loop from which deleteLater() was called.

thanks in advance
best regards
Tomasz

[ChrisW67]

Shows just how tolerant web browsers are.

You can use the same QNetworkAccessManager for the life of the program. You get a new QNetworkReply with each request.

Starting a request is a non-blocking operation, so several requests can be issued before any reply is received. If you store the reply pointer in a member variable (in NetRepl) and the second request is issued before the first is finished completely then you lose track of the first request. In that case you have a memory leak. Use the finished signal from QNetworkAccessManager then you need not keep a copy of the request pointer. The slot receives the reply pointer for the reply that just finished (they may not arrive in the order you issued them). You still need to arrange deletion.

[Tomasz]

So I should do something like this:

void MainWindow::getXML()
{
  QNetworkRequest request(QUrl("[my_address]"));
  NetAccMan.get(request);
  connect(&NetAccMan, SIGNAL(finished(QNetworkReply*)), this, SLOT(parseXML(QNetworkReply*)));
}
 
void MainWindow::parseXML(QNetworkReply *networkReply)
{
  [...]
 
  NetAccMan->disconnect();
  networkReply->deleteLater();
}

void MainWindow::getXML()
{
  QNetworkRequest request(QUrl("[my_address]"));
  NetAccMan.get(request);
  connect(&NetAccMan, SIGNAL(finished(QNetworkReply*)), this, SLOT(parseXML(QNetworkReply*)));
}

void MainWindow::parseXML(QNetworkReply *networkReply)
{
  [...]

  NetAccMan->disconnect();
  networkReply->deleteLater();
}

To copy to clipboard, switch view to plain text mode 

Would it be correct? Or deletion in this case should be arranged in different way?

thanks in advance
best regards
Tomasz

[ChrisW67]

Delete looks fine to me. You don't want to connect/disconnect your QNetworkAccessManager with every request though, just make the connection once in the MainWindow constructor and wait until the MainWindow is destroyed for automatic disconnection.