QNetworkReply | readyRead() cut by finished()

[AmirH]

Hi,

I coded to download a RSS file :

The problem is that the QByteArray seems to be truncated because of the code of the connect finished() -> deleteLater()
If I don't connect, qdebug() seems to have a weird behavior, I'm not sure what to do.

At term, I would like to be able to read the entire QByteArray (content of RSS) to fill a local file with some of the content, and it will be a thread loop, to netReply, and others, will be reused.

Here is my code :

void Window::requestRSSalaune(){
    QString urlRss = "http://rss.lemonde.fr/c/205/f/3050/index.rss";
 
    netRequest = QNetworkRequest(QUrl(urlRss));
    netReply = netMan->get(netRequest);
 
    QObject::connect (netReply, SIGNAL(readyRead()), this, SLOT(downloadedRSSalaune())) ;
 
    //QObject::connect (netReply, SIGNAL(finished()), netReply, SLOT(deleteLater())) ;
}
 
void Window::downloadedRSSalaune(){
 
    qDebug() << netReply->errorString();
 
    if(netReply->error() != QNetworkReply::NoError)
        return;
 
    QByteArray data =QByteArray(netReply->readAll());
 
 
    int statusCode = netReply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
    qDebug() << QVariant(statusCode).toString();
 
    if(data.isEmpty())
        return;
 
    qDebug()<<data;
}

void Window::requestRSSalaune(){
    QString urlRss = "http://rss.lemonde.fr/c/205/f/3050/index.rss";

    netRequest = QNetworkRequest(QUrl(urlRss));
    netReply = netMan->get(netRequest);

    QObject::connect (netReply, SIGNAL(readyRead()), this, SLOT(downloadedRSSalaune())) ;

    //QObject::connect (netReply, SIGNAL(finished()), netReply, SLOT(deleteLater())) ;
}

void Window::downloadedRSSalaune(){

    qDebug() << netReply->errorString();

    if(netReply->error() != QNetworkReply::NoError)
        return;

    QByteArray data =QByteArray(netReply->readAll());


    int statusCode = netReply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
    qDebug() << QVariant(statusCode).toString();

    if(data.isEmpty())
        return;

    qDebug()<<data;
}

To copy to clipboard, switch view to plain text mode 

First, connect finished is commented, here is the result :

"Unknown error"
"200"
"<?xml><rss><channel>[..]and lot of other words but during the sen"
"Unknown error"
"200"
"tence, qDebug() printed others things, as if the function was called another time during the print of data</guid></item></channel></rss>"

"Unknown error"
"200"
"<?xml><rss><channel>[..]and lot of other words but during the sen"
"Unknown error"
"200"
"tence, qDebug() printed others things, as if the function was called another time during the print of data</guid></item></channel></rss>"

To copy to clipboard, switch view to plain text mode 

Secondly, if I uncomment the connection finished() | deleteLater() :

"Unknown error"
"200"
"<?xml version='1.0' encoding='UTF-8'?>[...] and other words until this mome"
"Unknown error"
"200"

"Unknown error"
"200"
"<?xml version='1.0' encoding='UTF-8'?>[...] and other words until this mome"
"Unknown error"
"200"

To copy to clipboard, switch view to plain text mode 

So in this second case, the printing of data is not finished so seemed to be cut.
In the second case, when I uncomment, I have some execution where the qDebug() print is similar as when it's commented, it's very not regular.


So I don't really know how to manage it
Thanks in advance for your contribution.

[anda_skoa]

So in this second case, the printing of data is not finished so seemed to be cut.

If you are referring to the "mome" then this has likely been the end of the buffer as received in that slot call.
As you can see in the first example, the data continues in the next invocation, when the next readRead() signal is handled.

Cheers,
_

[AmirH]

ow, ok,
so how can I have all the content to analyse, shall I append the content of the reply in a new QFile?

and should I still use deleteLater() ? it seems to interompt the advancement of the download.

[anda_skoa]

so how can I have all the content to analyse

By reading all content whenever new content becomes available or by reading all content at the end of the operation.

shall I append the content of the reply in a new QFile?

A file is a good option if the amount of data can't fit into memory, etc.

and should I still use deleteLater() ?

If you want to free the resources associated with the network reply object. Most programs do.

Cheers,
_

[AmirH]

Hi,
I have another problem and I think it's no use to open a new thread.

here is the code modified the slot:

void Window::downloadedRSSalaune(){
 
    qDebug() << netReply->errorString();
 
    if(netReply->error() != QNetworkReply::NoError)
        return;
 
 
 
    QString filepath = QCoreApplication::applicationDirPath();
    filepath.append("\\alaune.xml");
    QFile *file = new  QFile();
    file->setFileName(filepath);
 
    qDebug()<<"hi1";
    if(!file->open(QIODevice::ReadWrite|QIODevice::Append|QIODevice::Text)){
        return;
    }
    qDebug()<<"hi2";
    file->resize(0);
    file->write(netReply->readAll());
qDebug()<<"hi3";
     QDomDocument *domfile = new QDomDocument("alaune_xml");
     QString errorStr;
         int errorLine;
         int errorColumn;
 
     if(!(domfile->setContent(file, false, &errorStr, &errorLine,
                              &errorColumn))){
         qDebug()<<errorStr;
         qDebug()<<errorLine;
         qDebug()<<errorColumn;
         return;
     }
     qDebug()<<"hi4";
    file->close();
 
   netReply->deleteLater();
 
   [...]
}

void Window::downloadedRSSalaune(){

    qDebug() << netReply->errorString();

    if(netReply->error() != QNetworkReply::NoError)
        return;



    QString filepath = QCoreApplication::applicationDirPath();
    filepath.append("\\alaune.xml");
    QFile *file = new  QFile();
    file->setFileName(filepath);

    qDebug()<<"hi1";
    if(!file->open(QIODevice::ReadWrite|QIODevice::Append|QIODevice::Text)){
        return;
    }
    qDebug()<<"hi2";
    file->resize(0);
    file->write(netReply->readAll());
qDebug()<<"hi3";
     QDomDocument *domfile = new QDomDocument("alaune_xml");
     QString errorStr;
         int errorLine;
         int errorColumn;

     if(!(domfile->setContent(file, false, &errorStr, &errorLine,
                              &errorColumn))){
         qDebug()<<errorStr;
         qDebug()<<errorLine;
         qDebug()<<errorColumn;
         return;
     }
     qDebug()<<"hi4";
    file->close();

   netReply->deleteLater();

   [...]
}

To copy to clipboard, switch view to plain text mode 

and the output :

"Unknown error"
hi1
hi2
hi3
"unexpected end of file"
1
1

"Unknown error"
hi1
hi2
hi3
"unexpected end of file"
1
1

To copy to clipboard, switch view to plain text mode 

so it return at setContent with the unexptected end of file for a file that, after I close the program, looks like this :

alaune.xml

<?xml version='1.0' encoding='UTF-8'?><?xml-stylesheet type='text/xsl' href='http://rss.lemonde.fr/xsl/fr/rss.xsl'?>
 
<rss xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" version="2.0"><channel>[A LOT OF THINGS]</channel></rss>

<?xml version='1.0' encoding='UTF-8'?><?xml-stylesheet type='text/xsl' href='http://rss.lemonde.fr/xsl/fr/rss.xsl'?>

<rss xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" version="2.0"><channel>[A LOT OF THINGS]</channel></rss>

To copy to clipboard, switch view to plain text mode 

i'm wondering if there is a unsynchronized method here. Whatever, where can be the problem ?

thank u for your help in advance

[anda_skoa]

I don't see where you reset the file's position to the beginning.

Also why do you write into a file at all?
Also why do you write into a file location that will almost certainly not be writeable once the application gets properly installed?
Also be aware that QDomDocument can not deal with partial documents.

Cheers,
_

[AmirH]

ok, I'm using QByteArray and it work very well, I've got articles titles with urls and the items menu work fine.

I'm using a file called history.xml that I fill every minutes and located here :

QString filepath = QCoreApplication::applicationDirPath();
filepath.append("\\history.xml");

what path should I define so it will be easy to access for my program when installed in program file via an installer?

[jefftee]

Look at QStandardPaths.

[AmirH]

I'm not really sure how to use it, is QStandardPaths:ataLocation the normal use.
In fact, I would like to know the common use of programs for files that are useless for the user to open it, but may not have to be very protected. What path to choose?

[jefftee]

Did you read the descriptions for each of the QStandardPaths::StandardLocation enums? There's a description for each that should guide you to a decision. That said, if I understand what you want, the following description of QStandardPaths::AppDataLocation seem to be a pretty good fit:

Returns a directory location where persistent application data can be stored. This is an application-specific directory. To obtain a path to store data to be shared with other applications, use QStandardPaths::GenericDataLocation

[AmirH]

Thank u for your advice.

Here's a problem that I cannot find any solution on web :

QString appPath;
    appPath = QString(QStandardPaths::​writableLocation(QStandardPaths::AppDataLocation));
    qDebug()<<appPath;

QString appPath;
    appPath = QString(QStandardPaths::​writableLocation(QStandardPaths::AppDataLocation));
    qDebug()<<appPath;

To copy to clipboard, switch view to plain text mode 

returrn :

stray '\342' in program
stray '\200' in program
stray '\213' in program

I checked all the '(' and '::', rewrite all the method by hand, cannot figure what's happening

[jefftee]

Have you by chance copied/pasted code from a web page somewhere? I would suggest selecting all of your code and copying to the clipboard, then paste into notepad or some other text editor where you can paste as plain text.

Once you do that, copy all of the plain text out of the notepad or text editor and paste as plain text into your source files and see if that resolves your problem.

[AmirH]

ok, thanks it works now, I have the printing of appDataLocation being AppData\Roaming\DirectNewsFrance

Here is the function to create history.xml (which will be modified a lot) by copying the model of historymodel.xml, which will be located in the program files folder of my app

void Window::createHistoryXml()
{
    qDebug()<<"hi1";
    QString filepath = QString(QStandardPaths::writableLocation(QStandardPaths::AppDataLocation));
    filepath.append("\\history.xml");
    QFileInfo checkFile(filepath);
    if (checkFile.exists() && checkFile.isFile()) {
            return;
        }
    qDebug()<<"hi2";
    AppFiles::historyXML->setFileName(filepath);
 
    QString modelpath = QCoreApplication::applicationDirPath();
    modelpath.append("\\historymodel.xml");
    QFile *modelXml = new QFile();
    modelXml->setFileName(modelpath);
    qDebug()<<"hi3";
    if(!(modelXml->open(QIODevice::ReadOnly))){
        return;
    }
    qDebug()<<"hi4";
    // QDomDocument to parse xml file
    QDomDocument *domfile = new QDomDocument("history_xml");
    if(!(domfile->setContent(modelXml))){
        return;
    }
    modelXml->close();
    qDebug()<<"hi5";
    QString write_doc = domfile->toString();
 
    if(!(AppFiles::historyXML->open(QIODevice::WriteOnly))){
        return;
    }
    qDebug()<<"hi6";
    AppFiles::historyXML->resize(0);
    QTextStream stream(AppFiles::historyXML);
    stream << write_doc;
    AppFiles::historyXML->close();
    qDebug()<<"hi7";
}

void Window::createHistoryXml()
{
    qDebug()<<"hi1";
    QString filepath = QString(QStandardPaths::writableLocation(QStandardPaths::AppDataLocation));
    filepath.append("\\history.xml");
    QFileInfo checkFile(filepath);
    if (checkFile.exists() && checkFile.isFile()) {
            return;
        }
    qDebug()<<"hi2";
    AppFiles::historyXML->setFileName(filepath);

    QString modelpath = QCoreApplication::applicationDirPath();
    modelpath.append("\\historymodel.xml");
    QFile *modelXml = new QFile();
    modelXml->setFileName(modelpath);
    qDebug()<<"hi3";
    if(!(modelXml->open(QIODevice::ReadOnly))){
        return;
    }
    qDebug()<<"hi4";
    // QDomDocument to parse xml file
    QDomDocument *domfile = new QDomDocument("history_xml");
    if(!(domfile->setContent(modelXml))){
        return;
    }
    modelXml->close();
    qDebug()<<"hi5";
    QString write_doc = domfile->toString();

    if(!(AppFiles::historyXML->open(QIODevice::WriteOnly))){
        return;
    }
    qDebug()<<"hi6";
    AppFiles::historyXML->resize(0);
    QTextStream stream(AppFiles::historyXML);
    stream << write_doc;
    AppFiles::historyXML->close();
    qDebug()<<"hi7";
}

To copy to clipboard, switch view to plain text mode 

no files neither folder under AppData Roaming have been created. Is it something I forgot to do?

I have printed :

hi1
hi2
hi3
hi4
hi5

[anda_skoa]

You probably don't have the path yet: QDir::mkpath()

Also that looks like a rather wasteful approach to copy a file: QFile::copy()

But at least don't leak objects! Currently you unnecessarily allocate "modelXml" and "domfile" on the heap and forget to delete them. Just allocate on the stack.

Cheers,
_

[AmirH]

ok thanks, I will copy file and mkdir,
I always forget to delete allocated objects, I'm always figuring it will be delete at the end of function automaticaly. Maybe I should use make_unique<>. But here on the stack is ok.

[jefftee]

no files neither folder under AppData Roaming have been created. Is it something I forgot to do?

Yes, if you read the documentation, it states that the path returned may not exist, so you may have to create the directory.