HTTPS request returns damaged HTML body

[ljuhrich]

Hi there,

I recently wanted to write a tool exploring the data which is requestable in facebook's Graph API. However, I created a tool performing a minimal request of the user's id and name.

The code is as following:

void MainWindow::doRequest()
{
    // Read username, access token
    QString id = QString(ui->edUserID->text());
    QString access_token = QString(ui->edAcessToken->text());
 
    socket = new QSslSocket(this);
    QString data = "GET https://graph.facebook.com/"
            + id
            + "?fields=id,name,about&method=GET&format=json&access_token="
            + QUrl::toPercentEncoding( access_token )
            + " HTTP/1.1\r\n"
            + "Host: graph.facebook.com\r\n"
            + "Connection: Keep-alive\r\n"
            + "Accept-Encoding: gzip,deflate\r\n"
            + "User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20100101 Firefox/21.0\r\n"
            + "Accept: */*\r\n"
            + "\r\n"
            ;
 
    connect(socket, SIGNAL(readyRead()), this, SLOT(resp()));
 
 
    socket->connectToHostEncrypted("graph.facebook.com", 443);
    if (!socket->waitForEncrypted())
    {
        ui->edResponse->setText(socket->errorString());
    }
 
    socket->write(data.toAscii());
 
}
 
void MainWindow::resp()
{
    QString responseData;
    QString tmp = QString(socket->readAll().data());
    responseData += tmp;
 
    ui->edResponse->setText( ui->edResponse->toPlainText() + "\r\n" +  responseData);
}

void MainWindow::doRequest()
{
    // Read username, access token
    QString id = QString(ui->edUserID->text());
    QString access_token = QString(ui->edAcessToken->text());

    socket = new QSslSocket(this);
    QString data = "GET https://graph.facebook.com/"
            + id
            + "?fields=id,name,about&method=GET&format=json&access_token="
            + QUrl::toPercentEncoding( access_token )
            + " HTTP/1.1\r\n"
            + "Host: graph.facebook.com\r\n"
            + "Connection: Keep-alive\r\n"
            + "Accept-Encoding: gzip,deflate\r\n"
            + "User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20100101 Firefox/21.0\r\n"
            + "Accept: */*\r\n"
            + "\r\n"
            ;

    connect(socket, SIGNAL(readyRead()), this, SLOT(resp()));


    socket->connectToHostEncrypted("graph.facebook.com", 443);
    if (!socket->waitForEncrypted())
    {
        ui->edResponse->setText(socket->errorString());
    }

    socket->write(data.toAscii());

}

void MainWindow::resp()
{
    QString responseData;
    QString tmp = QString(socket->readAll().data());
    responseData += tmp;

    ui->edResponse->setText( ui->edResponse->toPlainText() + "\r\n" +  responseData);
}

To copy to clipboard, switch view to plain text mode 

With that code i get a response as following:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Type: application/json; charset=UTF-8
ETag: "581d7bde213d9c05777d0b6355eb626070e238fe"
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
X-FB-Rev: 831988
X-FB-Debug: GGFSORJxReaKTcVhWmvBe6iBZoQTLyjEFm++YDBAGwM=
Date: Fri, 31 May 2013 18:11:07 GMT
Connection: keep-alive
Content-Length: 58


{
"id": "100002634417763",
"name": "Lukas Juhrich"
}

So far, so well.
But when I add some property fields, e.g. “sports,likes,interests” (added right after the “about” in the request string), the response gets damaged, although the very same request made by facebook's graph-explorer tool returned a correct result:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: private, no-cache, no-store, must-revalidate
Content-Type: application/json; charset=UTF-8
ETag: "d8e5cc3516f63acfeebc1a4db5b4a56b92809b7c"
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
X-FB-Rev: 831988
Content-Encoding: gzip
X-FB-Debug: mm/FphJrwWeme7wz/uyv2FiadQxR6TdJXA//z7xgTnw=
Date: Fri, 31 May 2013 18:14:51 GMT
Connection: keep-alive
Content-Length: 10612


‹
¬Ýtôµéªszð¾c]¾ÙûéêøÍ{•
^
ÃŒîʓ+ï±µ’7;Q•ô� �ŒKk}É8AUHHHQ$õùú÷oµªT• luTA`Ã@"þ›È8‘©Ò ¤T‡x³eÂŽpêà :ûq‘S÷Â� �{ÆÏÚ¹¤n+ÝÙ7ªÀÂ� �Š&•ïT»5´`*¡/•‰+˜µ[h$ÔúŽ£q9öël¥ŽÂŒ „úMAvKäN´’裧Â� ��S¹ÔyQö<¡¼
Á1—•Ã� �ç—láªZµß‚æª`Â� ��z$`$QôâöÅ)wn!Ôdˆ§‹K W`§¯“Y¾·®t{?½ŸŒÒ� �¨gÃ*Ÿ\¼Škþ„i_-V=DX±˜W¬«V:¬š,#ò¦™ ί.ë+“h£BUÖ}¿„â2-ŹÎã{•ëéêþZ¨à –è îK3¡¾ÞX¿LAªÛî’x‰ >ºæjv9Èâ
¾<©Šµïî0Yª£Ge‘¶P4 ’ØQãâ}´¬¦
wCC4ƒLƍïˆ
`ºs´ïW“4¥Ú*T#綁0¾nôQœÊ1Ó8nm¨ó%Ê#Å)%ß"…nB7� ��Þ\=¬óxQ
£Stûg%«ªæ ¤a
£)¨ya²ÆóQ“£´vÇê #`+EyñÝTs!Eç!Ã*H—¦¿² ?<AÓ¤bM¹¢“š»Tµæ h~R¥ŠEµ©%ýŠ×»4nßà ´>Ã’Z÷N~}«$ \®º#kÆó1#WjÃœ™V¾ÓÃ� �øÝ1
ô‹þzÂ’†ù*ªÿV¾ûˆ6"¢m†\Ã� �(ÂŽsßÀ4Éՙ9Ã*¿©U ¾ÑÅ:ϖn2,:ää¦fEüb ·Âœ¯¼Önُ¡!N ºö)ˆ'ž šTÿö¡• ñ
*(SP©³¥8ZSöÄfKf:°£qDÃ� �82FžÀ5ø$SØI|0ò°Ó¤ —.sdiƲ'Z!ê^!®Ø¯€ ÀÖ<ÚmÀ¯ì‹T«$-氅jÀ¯8©ËâQ ÞHS_ÍÕo;‰–=¦Øy:Ù¹ £d;É*4 ×õ°VOmv°¿z§ÐrZÀÃ� ��$Â¥z˜^’Úæ¦<(¢dJˆ¨;(  Ãœ_Róª<’”T7b)5o)¿Ì)T ª)ñW3öîHö*ÞÍ
·Ã*ÿ Q P œõfÝJ”jR½hn£j‚Ä\ ¦ÖU”¡ð¢8n Mç0î)Í(‘ÓÛ•# azȢF êØ*8û&"Ã¥w7}R1–Pý¦ÓÍ ^asèåóÃ*ÔB¨ÍʬÈ4}AÊKÃ*à ” ’ÝNðPbÕ=¦â˜‘wm˜oÃ� �ºczHGÃ’~ú3Ã’2Yš©nb6J€3V66à ‘ey¥†,ç•ÓèöÙ.Ã� �µ<˜··H©]•Ö®ñxByžÊF6$óAÃŒJMÃ� �ÂŽpW¹ìÒêÂ4û¡+ý†yG y膎õª2'É;Ñò[H¡¡&õ*êÂHy
–ÞôòÝ@?Ã*äs:±®‰¾ šB,*)£«À·t%žtPR=U*Ë tÆOãwu6j<ò«·=ùŽ¡üE� �“ÁY×

Do you have any kind of idea / tips etc. for me? thank you!

[wysota]

One tip I can definitely give you is to not expect all data to come in one chunk. You can see that content-length header's value is 10612 which is definitely more than what you posted. Another tip I might give you is that if a header says "content-encoding: gzip" then it means content is compressed and you need to uncompress it.

[anda_skoa]

Also might make sense to check out QNetworkAccessManager to handle all the HTTP level (and below) protocol for you.

Cheers,
_

[ljuhrich]

NICE! It works!

Thank you very much!

both tips were very useful:

1. After I changed Accpt–encoding to none, I got some readable text, but not everything.
2. But when I changed the code using a QNetworkAcessmanager, it returned everything, and the code is a lot simpler.

void MainWindow::doRequest()
{
    // INITIALIZATION
 
    // Read username, access token
    QString id = QString(ui->edUserID->text());
    QString access_token = QString(ui->edAcessToken->text());
 
 
    manager = new QNetworkAccessManager(this);
    connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(resp(QNetworkReply*)));
 
    QNetworkRequest request(
                "https://graph.facebook.com/"
                + id
                + "?fields=id,name,about,sports,likes,interests&method=GET&format=json&access_token="
                + QUrl::toPercentEncoding( access_token )
                );
    request.setRawHeader("Accept-Encoding", "none");
    request.setRawHeader("User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20100101 Firefox/21.0");
 
    manager->get(request);
}
 
void MainWindow::resp(QNetworkReply *reply)
{
    QString responseData = QString::fromAscii(reply->readAll());
    ui->edResponse->setText( ui->edResponse->toPlainText() + "\r\n" +  responseData);
}

void MainWindow::doRequest()
{
    // INITIALIZATION

    // Read username, access token
    QString id = QString(ui->edUserID->text());
    QString access_token = QString(ui->edAcessToken->text());


    manager = new QNetworkAccessManager(this);
    connect(manager, SIGNAL(finished(QNetworkReply*)), this, SLOT(resp(QNetworkReply*)));

    QNetworkRequest request(
                "https://graph.facebook.com/"
                + id
                + "?fields=id,name,about,sports,likes,interests&method=GET&format=json&access_token="
                + QUrl::toPercentEncoding( access_token )
                );
    request.setRawHeader("Accept-Encoding", "none");
    request.setRawHeader("User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:21.0) Gecko/20100101 Firefox/21.0");

    manager->get(request);
}

void MainWindow::resp(QNetworkReply *reply)
{
    QString responseData = QString::fromAscii(reply->readAll());
    ui->edResponse->setText( ui->edResponse->toPlainText() + "\r\n" +  responseData);
}

To copy to clipboard, switch view to plain text mode 

I wish you a nice day

[anda_skoa]

Just in case: one thing people who are new to using QNetworkAccessManager can miss is that it can be used for as many requests in parallel as one would want.
I.e. in your code you create the object every time doRequest() is called, but you would only have to do it once.

Each request is paired with a reply object and QNAM is able to process them individually, even multiple requests at the same time.

Cheers,
_

[ljuhrich]

Thnk you for these nice tips, I always try to improve my skills — so every little tip helps me a lot and saves me from hours of debugging

[anda_skoa]

Well, then I have another one

You will need to delete the reply object, otherwise you are leaking it.
One way to do that is to schedule its deletion in the slot connected to the finished signals

void MainWindow::resp(QNetworkReply *reply)
{
    QString responseData = QString::fromAscii(reply->readAll());
    ui->edResponse->setText( ui->edResponse->toPlainText() + "\r\n" +  responseData);
 
    reply->deleteLater(); // schedule deletion when event loop continues event processing
}

void MainWindow::resp(QNetworkReply *reply)
{
    QString responseData = QString::fromAscii(reply->readAll());
    ui->edResponse->setText( ui->edResponse->toPlainText() + "\r\n" +  responseData);

    reply->deleteLater(); // schedule deletion when event loop continues event processing
}

To copy to clipboard, switch view to plain text mode 

Cheers,
_