QString static callback function from CURL

[tpf80]

This piece of code tells the CURL library where to put what its gathering:

curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);

curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);

To copy to clipboard, switch view to plain text mode 

This piece of code tells it the callback function to use:

curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);

curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);

To copy to clipboard, switch view to plain text mode 

Once you execute curl exec it does whatever the options tell it to do, and calls the callback function whenever the remote server sends back data. Im pretty sure its safe as copying the pointer is what the examples in the CURL install tarball show, and I haven't had any problems with it.

The main problem I face is that I am pulling in several megabytes of data, which normally would append to the data already in memory from the request. Rather than waste performance on pulling all of that data into memory I wanted to be able to look at the chunks as they come in, pull out what I want out of them, and pretty much throw away the rest.

Since the callback function is a static function, I am not sure how I can make the data that is processed in it available to my QT app outside of that function.

I did try the static member approach as marcel suggested, which compiled fine, however the program crashed when it got to the point that it started using that variable. Do you have an example of how this would work?

[wysota]

The main problem I face is that I am pulling in several megabytes of data, which normally would append to the data already in memory from the request. Rather than waste performance on pulling all of that data into memory I wanted to be able to look at the chunks as they come in, pull out what I want out of them, and pretty much throw away the rest.

So why do you use realloc for that? And is using curl really necessary? Do you use some specific options that QHttp can't handle?

Since the callback function is a static function, I am not sure how I can make the data that is processed in it available to my QT app outside of that function.

Through global variables or static members (in a singleton class).

I did try the static member approach as marcel suggested, which compiled fine, however the program crashed when it got to the point that it started using that variable. Do you have an example of how this would work?

Something like this should work:

class S {
public:
  S *instance(){
    static S* inst = new S;
    return inst;
  }
  const char *getPtr() const { return S::staticPtr; }
  static void callBack(){
  //...
  staticPtr = ...;
  }
private:
  static char *staticPtr;
};
char *S::staticPtr = 0;

class S {
public:
  S *instance(){
    static S* inst = new S;
    return inst;
  }
  const char *getPtr() const { return S::staticPtr; }
  static void callBack(){
  //...
  staticPtr = ...;
  }
private:
  static char *staticPtr;
};
char *S::staticPtr = 0;

To copy to clipboard, switch view to plain text mode 

[patrik08]

You dont need callback if you set a time limit! on curl!


If you need callback use http://doc.trolltech.com/4.0/qhttp.html

inside ..
http://sourceforge.net/projects/qtexcel-xslt/
you find more sample from curl ...

/* return file contenets as qstring  from local or remote file ... like php*/
QString curl_get_contents( QString fullFileName )
{
        QString inside = "";
        qDebug() << "###1 curl_get_contents   "<< fullFileName;
        QString xdcookiefile, xml_export_file, wwwnetfile;
        if (fullFileName.contains("https://", Qt::CaseInsensitive)) { 
        return inside;
        } 
        Init();
        qDebug() << "###2 curl_get_contents   "<< fullFileName;
 
          xdcookiefile = QString( "%1biscotti.html" ).arg( WORK_CACHEDIR );
          xml_export_file = QString( "%1_xml_export.xml" ).arg( WORK_CACHEDIR );
          wwwnetfile = QString( "%1wwwfull.html" ).arg( WORK_CACHEDIR );
 
       if (IsNetFile( fullFileName ) )
       {
        qt_unlink(wwwnetfile);
           qDebug() << "###3 curl_get_contents   "<< fullFileName;
         /*QString xgeturl = fullFileName;  http://shop.ecoplanet.ch/info.php */
           QByteArray lop = wwwnetfile.toAscii();
         char *localfile = lop.data();
           QByteArray der = xdcookiefile.toAscii();
         char *xwcookiefile = der.data();
          QByteArray ba = fullFileName.toAscii();
         char *url = ba.data();
         qDebug() << "### entra save to   "<< localfile;
         qDebug() << "### get url  "<< url;
            CURL *curl_handle;
            FILE *outfile;
            curl_global_init(CURL_GLOBAL_ALL);     
            curl_global_init(CURL_GLOBAL_ALL);
            curl_handle = curl_easy_init();
            outfile = fopen(localfile, "w"); 
 
                if (outfile!=NULL) {
                    /* CURLOPT_COOKIE and must have CURLOPT_COOKIEJAR char * (file to write same as php)   */
                    curl_easy_setopt(curl_handle, CURLOPT_URL, url);
                    curl_easy_setopt(curl_handle, CURLOPT_VERBOSE, TRUE);
                    curl_easy_setopt(curl_handle, CURLOPT_COOKIEJAR, xwcookiefile );
                    curl_easy_setopt(curl_handle, CURLOPT_COOKIEFILE, xwcookiefile );
                    curl_easy_setopt(curl_handle, CURLOPT_FOLLOWLOCATION , 1);
                     #if defined Q_WS_MAC
                    curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Mac QT4 / PPK_W @ciz.ch" );
                    #endif
                    #if defined(Q_WS_WIN)
                    curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Windows QT4 / PPK_W @ciz.ch" );
                    #endif
                    curl_easy_setopt(curl_handle, CURLOPT_MAXREDIRS , 2);
                    curl_easy_setopt(curl_handle, CURLOPT_TIMEOUT , 20 );
                    curl_easy_setopt(curl_handle, CURLOPT_FILE, outfile);   
 
                     if (curl_easy_perform(curl_handle)==CURLE_OK) {
                         fclose(outfile);
 
                             if (is_file(wwwnetfile)) {
                                          inside = ReadFile( wwwnetfile ); 
                                          if (inside.size() > 1) {
                                          ERROR_MSG = "";
                                          #if defined Q_WORKS_PEND
                                          /* not delete file */
                                          #else
                                          qt_unlink(wwwnetfile);
                                          #endif
                                          }
                              }
 
                     }  else {
                     ERROR_MSG = "Error time out to get remote file";
                     }                         
                } 
       /* return grab result from local file */
      return inside;
      }
 
 
 
 
    /*  ok is a fake normal local file init .....  */
    inside = ReadFile( fullFileName ); 
    return inside;
}

/* return file contenets as qstring  from local or remote file ... like php*/
QString curl_get_contents( QString fullFileName )
{
        QString inside = "";
        qDebug() << "###1 curl_get_contents   "<< fullFileName;
        QString xdcookiefile, xml_export_file, wwwnetfile;
        if (fullFileName.contains("https://", Qt::CaseInsensitive)) { 
        return inside;
        } 
        Init();
        qDebug() << "###2 curl_get_contents   "<< fullFileName;
        
          xdcookiefile = QString( "%1biscotti.html" ).arg( WORK_CACHEDIR );
          xml_export_file = QString( "%1_xml_export.xml" ).arg( WORK_CACHEDIR );
          wwwnetfile = QString( "%1wwwfull.html" ).arg( WORK_CACHEDIR );
        
       if (IsNetFile( fullFileName ) )
       {
        qt_unlink(wwwnetfile);
           qDebug() << "###3 curl_get_contents   "<< fullFileName;
         /*QString xgeturl = fullFileName;  http://shop.ecoplanet.ch/info.php */
           QByteArray lop = wwwnetfile.toAscii();
         char *localfile = lop.data();
           QByteArray der = xdcookiefile.toAscii();
         char *xwcookiefile = der.data();
          QByteArray ba = fullFileName.toAscii();
         char *url = ba.data();
         qDebug() << "### entra save to   "<< localfile;
         qDebug() << "### get url  "<< url;
            CURL *curl_handle;
            FILE *outfile;
            curl_global_init(CURL_GLOBAL_ALL);     
            curl_global_init(CURL_GLOBAL_ALL);
            curl_handle = curl_easy_init();
            outfile = fopen(localfile, "w"); 
 
                if (outfile!=NULL) {
                    /* CURLOPT_COOKIE and must have CURLOPT_COOKIEJAR char * (file to write same as php)   */
                    curl_easy_setopt(curl_handle, CURLOPT_URL, url);
                    curl_easy_setopt(curl_handle, CURLOPT_VERBOSE, TRUE);
                    curl_easy_setopt(curl_handle, CURLOPT_COOKIEJAR, xwcookiefile );
                    curl_easy_setopt(curl_handle, CURLOPT_COOKIEFILE, xwcookiefile );
                    curl_easy_setopt(curl_handle, CURLOPT_FOLLOWLOCATION , 1);
                     #if defined Q_WS_MAC
                    curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Mac QT4 / PPK_W @ciz.ch" );
                    #endif
                    #if defined(Q_WS_WIN)
                    curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "Windows QT4 / PPK_W @ciz.ch" );
                    #endif
                    curl_easy_setopt(curl_handle, CURLOPT_MAXREDIRS , 2);
                    curl_easy_setopt(curl_handle, CURLOPT_TIMEOUT , 20 );
                    curl_easy_setopt(curl_handle, CURLOPT_FILE, outfile);   
                    
                     if (curl_easy_perform(curl_handle)==CURLE_OK) {
                         fclose(outfile);
                         
                             if (is_file(wwwnetfile)) {
                                          inside = ReadFile( wwwnetfile ); 
                                          if (inside.size() > 1) {
                                          ERROR_MSG = "";
                                          #if defined Q_WORKS_PEND
                                          /* not delete file */
                                          #else
                                          qt_unlink(wwwnetfile);
                                          #endif
                                          }
                              }
                             
                     }  else {
                     ERROR_MSG = "Error time out to get remote file";
                     }                         
                } 
       /* return grab result from local file */
      return inside;
      }

    
    
     
    /*  ok is a fake normal local file init .....  */
    inside = ReadFile( fullFileName ); 
    return inside;
}

To copy to clipboard, switch view to plain text mode 

[tpf80]

So why do you use realloc for that? And is using curl really necessary? Do you use some specific options that QHttp can't handle?

I used that in the first place because thats what the example from CURL themselves used. For my purposes, since I don't need the whole data chunk that is pulled in, I've removed that part and use pretty much this for making the data within the callback usable:

resData = QString::fromUtf8 ((char *)ptr);

resData = QString::fromUtf8 ((char *)ptr);

To copy to clipboard, switch view to plain text mode 

As for why I was using CURL instead of QHttp, It was mainly because I didn't see QHttp supporting HTTPS (at least in the free edition that I have at the moment) and I was already pretty familiar with CURL's functionality as a standalone program and using it in PHP. This is my first attempt at using it with C++ though.

My problem is probably related to my unfamiliarity to using a variable in the way I need to. The static member code that you posted looks very close to what I had, I'm taking a look at it and see what I was missing.

[tpf80]

Actually, I found out that I might not need https for what I am doing. I think I'll try out QHttp and see if that doesn't solve my problem, rather than making a mess of my program trying to force it to work with curl.

[wysota]

As for why I was using CURL instead of QHttp, It was mainly because I didn't see QHttp supporting HTTPS (at least in the free edition that I have at the moment)

Qt 4.3+ supports SSL sockets. The release candidate of Qt 4.3 is already available at TT site.

[tpf80]

I also notice that QHttp doesn't really have a good way to do cookies like curl does.
It seems like In order to keep my session, I would need to extract the cookie after each request, and then add it to the next one in a semi-manual fashion.

When I use curl, I just can issue a command which creates a "cookiejar" in memory when i first start it up and handles preserving the cookies itself.

So I guess I kindof have to decide what I want to do:

1) use CURL and figure out this static member variable stuff to get the data I want out (since I already have curl logging in, doing cookies for the session, etc perfectly already)

or..

2) rewrite code in QHTTP which would feel cleaner but would have to worry about a few more things like the cookies myself.

[wysota]

I also notice that QHttp doesn't really have a good way to do cookies like curl does.
It seems like In order to keep my session, I would need to extract the cookie after each request, and then add it to the next one in a semi-manual fashion.

Not exactly (I use a base request header object where I set the cookie, so that I don't have to do it on every request), but in general you're correct.

When I use curl, I just can issue a command which creates a "cookiejar" in memory when i first start it up and handles preserving the cookies itself.

Curl is definitely a higher level tool than QHttp so it has much much more capabilities (QHttp is really only an interface to the protocol), but on the other hand it'd be much easier to parse the result using QHttp.

[tpf80]

Thanks to all that replied, in the end I used libcurl for what I was doing with the HTTP stuff, and got it to work real well in fact.

What I ended up doing was using the std string/char functions in the callback function, so that it would take out what it needed on the fly and write it to the memory chunk. Since in the callback its using those, it made things easier and have much better performance.

In the main Qt program, I used the QString, by appending the returned data chunk via QString::fromUtf8. I could then do with it what I wanted to.

[tpf80]

You dont need callback if you set a time limit! on curl!


If you need callback use http://doc.trolltech.com/4.0/qhttp.html

inside ..
http://sourceforge.net/projects/qtexcel-xslt/
you find more sample from curl ...

In the code that you wrote it looks like curl is writing to a file. How would I use it without a callback function if I wanted to pull the data into memory only and not write anything to a file?