multi threading or QConcurrent for the scenario

[ejoshva]

Hi,

I have a scenario where in I have to show a document on click of the document icon. Each page of the document is a separate html which is encrypted.
Due to the huge number of pages, it take more time to decrypt all the html files and show the page.

Kindly suggest which one is the best suitable (QThread / QConcurrent) , so that I can show the page once the 1st file is decrypted without waiting for all files to complete decrypting process. I want to make the page viewing and the decrypting happen simultaneously once the decrypting of the 1st file is done.

Thanks in advance

[Santosh Reddy]

As you don't need any background activity to be done continuously, it would be better to use QConcurrent, just run the decrypting function QtConcurrent::run().

[ejoshva]

Thanks for the quick response.

[wysota]

I would rather put all pages into a container and use QtConcurrent::map().

[ejoshva]

How do I use QtConcurrent::map()

I have tried to implement QtConcurrent, but it crashes
stating the below error
"Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff8cb0f866 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff87aee35c pthread_kill + 92
2 libsystem_c.dylib 0x00007fff89f2fb1a abort + 125
3 org.qt-project.QtCore 0x000000010c1a2159 qt_message_fatal(QtMsgType, QMessageLogContext const&, QString const&) + 9
4 org.qt-project.QtCore 0x000000010c1a3611 QMessageLogger::fatal(char const*, ...) const + 161
5 org.qt-project.QtCore 0x000000010c19e815 qt_assert_x(char const*, char const*, char const*, int) + 85
6 com.yourcompany.learnOn 0x000000010b2cd880 QList<QFileInfo>:perator[](int) + 96 (qlist.h:486)
7 com.yourcompany.learnOn 0x000000010b2cbf8a loadbook::decryptLoop(int, int) + 138 (loadbook.cpp:122)

ASSERT failure in QList<T>:perator[]: "index out of range", file /Applications/QT/5.4/clang_64/lib/QtCore.framework/Headers/qlist.h, line 486
The program has unexpectedly finished.


The code is as below

totalXHtmlPages = 173
    int index=0;
 
    while( index <totalXHtmlPages)
    {
        if(stopThread) return;
 
 
        QFuture<QStringList> test = QtConcurrent::run(this,&loadbook::decryptLoop,index,index+13);
        PgContents.append(test.result());
 
        index = index + 13;
    }
 
 
QStringList loadbook::decryptLoop(int startIndex, int endIndex)
{
    QStringList decryptContent;
    for(int index = startIndex; index < endIndex; index++)
    {
 
        if(stopThread) break;
        decryptContent.append(decrypt->decryptFile(FileInfoList[index].absoluteFilePath(), 'A'));
        qDebug()<<"FileInfoList["<<index<<"] -> "<<FileInfoList[index].absoluteFilePath();
    }
    return decryptContent;
}

totalXHtmlPages = 173
    int index=0;

    while( index <totalXHtmlPages)
    {
        if(stopThread) return;
     

        QFuture<QStringList> test = QtConcurrent::run(this,&loadbook::decryptLoop,index,index+13);
        PgContents.append(test.result());

        index = index + 13;
    }


QStringList loadbook::decryptLoop(int startIndex, int endIndex)
{
    QStringList decryptContent;
    for(int index = startIndex; index < endIndex; index++)
    {

        if(stopThread) break;
        decryptContent.append(decrypt->decryptFile(FileInfoList[index].absoluteFilePath(), 'A'));
        qDebug()<<"FileInfoList["<<index<<"] -> "<<FileInfoList[index].absoluteFilePath();
    }
    return decryptContent;
}

To copy to clipboard, switch view to plain text mode 

Here I dont find multiple thread which I intended to create.

how can I make this 173 files to be decrypted as fast as possible using threads

[wysota]

Your code makes no sense. You start a thread only to wait until it finishes executing. Use QtConcurrent::map() as advised.

[ejoshva]

QList<QString> fileList;
    for(int j=1;j<=settings->childKeys().count();j++)
    {
        QString temp = bkpath+settings->value(QString::number(j)).toString();
        fileList.append(temp);
    }
 QFuture<QStringList> test = QtConcurrent::map(fileList,&Decrypt::decryptFile));
 
 
QString Decrypt::decryptFile(QString sourceFile,QChar keystring)
{
    _decrptedResStr.clear();
    string srcFile=sourceFile.toUtf8().constData();
    ifstream ffin;
 
    ffin.open(srcFile.c_str(),ios_base::in|ios_base::binary);
 
    char keyToEncryption= keystring.toLatin1();
    if(ffin.is_open())
    {
        char ch;
        while(!(ffin.read(&ch,1).eof()))
        {
            ch ^= keyToEncryption;
            _decrptedResStr.append(ch);
        }
    }
    ffin.close();
    return _decrptedResStr;
}

QList<QString> fileList;
    for(int j=1;j<=settings->childKeys().count();j++)
    {
        QString temp = bkpath+settings->value(QString::number(j)).toString();
        fileList.append(temp);
    }
 QFuture<QStringList> test = QtConcurrent::map(fileList,&Decrypt::decryptFile));


QString Decrypt::decryptFile(QString sourceFile,QChar keystring)
{
    _decrptedResStr.clear();
    string srcFile=sourceFile.toUtf8().constData();
    ifstream ffin;

    ffin.open(srcFile.c_str(),ios_base::in|ios_base::binary);

    char keyToEncryption= keystring.toLatin1();
    if(ffin.is_open())
    {
        char ch;
        while(!(ffin.read(&ch,1).eof()))
        {
            ch ^= keyToEncryption;
            _decrptedResStr.append(ch);
        }
    }
    ffin.close();
    return _decrptedResStr;
}

To copy to clipboard, switch view to plain text mode 

But I get the error
/Users/user/LearnOn/loadbook.cpp:171: error: no matching function for call to 'blockingMapped'
QFuture<QStringList> test = QtConcurrent::map(fileList,&Decrypt::decryptFile)) ;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~


(My bad posted the wrong function , edited it )

[anda_skoa]

That doesn't make much sense.
Your list elements are of type QString, but your map function takes QChar.

Cheers,
_

[ejoshva]

QList<QString> test = QtConcurrent::map(fileList,&Decrypt::decryptLoop);
 
QString loadbook::decryptLoop(QString fileName)
{
    return  decrypt->decryptFile(fileName, 'A');;
}

QList<QString> test = QtConcurrent::map(fileList,&Decrypt::decryptLoop);

QString loadbook::decryptLoop(QString fileName)
{
    return  decrypt->decryptFile(fileName, 'A');;
}

To copy to clipboard, switch view to plain text mode 

Even after modifying like this , I am still getting the same issue.
/Users/user/LearnOn/loadbook.cpp:163: error: no matching function for call to 'map'
QList<QString> test = QtConcurrent::map(fileList,&Decrypt::decryptFile);
^~~~~~~~~~~~~~~~~

I am new to QT, kindly guide me to resolve this issue.

[wysota]

Your decryptFile() method is not re-entrant.

[ejoshva]

how do I make it re-entrant?

[wysota]

http://en.wikipedia.org/wiki/Reentrancy_(computing)

[anda_skoa]

Even after modifying like this , I am still getting the same issue.
/Users/user/LearnOn/loadbook.cpp:163: error: no matching function for call to 'map'
QList<QString> test = QtConcurrent::map(fileList,&Decrypt::decryptFile);
^~~~~~~~~~~~~~~~~

At least in your code snippet shown here you are still passing the wrong function.
Also the return value of map() is QFuture<void>

Cheers,
_

[ejoshva]

QString loadbook::decryptLoop(QString &fileName)
{
    QString content;
    content = decrypt->decryptFile(fileName, 'A');
    return content;
}
 
QList<QString> test = QtConcurrent::map(fileList,&loadbook::decryptLoop);

QString loadbook::decryptLoop(QString &fileName)
{
    QString content;
    content = decrypt->decryptFile(fileName, 'A');
    return content;
}

QList<QString> test = QtConcurrent::map(fileList,&loadbook::decryptLoop);

To copy to clipboard, switch view to plain text mode 

Hope this is re-entrant.

now the errors are
/Users/user/LearnOn/loadbook.cpp:159: error: no viable conversion from 'QFuture<void>' to 'QList<QString>'
QList<QString> test = QtConcurrent::map(fileList,&loadbook::decryptLoop) ;
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/QT/5.4/clang_64/lib/QtConcurrent.framework/Headers/qtconcurrentmapkernel.h:63: error: no matching function for call to object of type 'QtConcurrent::MemberFunctionWrapper1<QString, loadbook, QString &>'
map(*it);
^~~

---

Added after 28 minutes:

I tried with mappedReduced

QFuture<QString> test = QtConcurrent::mappedReduced(fileList,&loadbook::decryptLoop,&loadbook::joinContent);
         test.waitForFinished();
 
QString loadbook::decryptLoop(QString &fileName)
{
    QString content;
    content = decrypt->decryptFile(fileName, 'A');
    return content;
}
void joinContent(QString &reduceResult, const QString &partial) {
    reduceResult += partial;
}

QFuture<QString> test = QtConcurrent::mappedReduced(fileList,&loadbook::decryptLoop,&loadbook::joinContent);
         test.waitForFinished();

QString loadbook::decryptLoop(QString &fileName)
{
    QString content;
    content = decrypt->decryptFile(fileName, 'A');
    return content;
}
void joinContent(QString &reduceResult, const QString &partial) {
    reduceResult += partial;
}

To copy to clipboard, switch view to plain text mode 

getting the below error
/Users/user/LearnOn/loadbook.cpp:161: error: no matching function for call to 'mappedReduced'
QFuture<QString> test = QtConcurrent::mappedReduced(fileList,&loadbook::de cryptLoop,&loadbook::joinContent);
^~~~~~~~~~~~~~~~~~~~~~~~~~~

[wysota]

Hope this is re-entrant.

Depends what decryptFile() does. If you didn't change its code then it's not reentrant.

[ejoshva]

QString loadbook::decryptLoop(QString &fileName)
{
    QString content;
    content = decrypt->decryptFile(fileName, 'A');
    return content;
}
void joinContent(QString &reduceResult, const QString &partial) {
    reduceResult += partial;
}
 
     QFuture<QStringList> test = QtConcurrent::mappedReduced(fileList,&loadbook::decryptLoop,&loadbook::joinContent,QtConcurrent::SequentialReduce);
         test.waitForFinished();

QString loadbook::decryptLoop(QString &fileName)
{
    QString content;
    content = decrypt->decryptFile(fileName, 'A');
    return content;
}
void joinContent(QString &reduceResult, const QString &partial) {
    reduceResult += partial;
}

     QFuture<QStringList> test = QtConcurrent::mappedReduced(fileList,&loadbook::decryptLoop,&loadbook::joinContent,QtConcurrent::SequentialReduce);
         test.waitForFinished();

To copy to clipboard, switch view to plain text mode 

I have added a code like above so that I want all the files loaded in the fileList have to be decrypted simultaneously and the decrypted content are to be caught in the test.

When I try to compile this code getting the error as below

/Users/user/LearnOn/loadbook.cpp:142: error: no matching function for call to 'mappedReduced'
QFuture<QStringList> test = QtConcurrent::mappedReduced(fileList,&loadbook::de cryptLoop,&loadbook::joinContent,QtConcurrent::Seq uentialReduce);
^~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/QT/5.4/clang_64/lib/QtConcurrent.framework/Versions/5/Headers/qtconcurrentmap.h:115: candidate template ignored: couldn't infer template argument 'ResultType'
QFuture<ResultType> mappedReduced(const Sequence &sequence,
^
/Applications/QT/5.4/clang_64/lib/QtConcurrent.framework/Versions/5/Headers/qtconcurrentmap.h:142: candidate template ignored: deduced conflicting types for parameter 'Iterator' ('QList<QString>' vs. 'QString (loadbook::*)(QString &)')
QFuture<ResultType> mappedReduced(Iterator begin,
^
/Applications/QT/5.4/clang_64/lib/QtConcurrent.framework/Versions/5/Headers/qtconcurrentmap.h:156: candidate template ignored: deduced conflicting types for parameter 'Iterator' ('QList<QString>' vs. 'QString (loadbook::*)(QString &)')
QFuture<typename QtPrivate::ReduceResultType<ReduceFunctor>::Result Type> mappedReduced(Iterator begin,
^
/Applications/QT/5.4/clang_64/lib/QtConcurrent.framework/Versions/5/Headers/qtconcurrentmap.h:128: candidate template ignored: substitution failure [with Sequence = QList<QString>, MapFunctor = QString (loadbook::*)(QString &), ReduceFunctor = void (loadbook::*)(QString &, const QString &)]: implicit instantiation of undefined template 'QtPrivate::ReduceResultType<void (loadbook::*)(QString &, const QString &)>'
QFuture<typename QtPrivate::ReduceResultType<ReduceFunctor>::Result Type> mappedReduced(const Sequence &sequence,
~~~~~~~~~~~~~~~~ ^

Kindly help to resolve this issue. I have been sitting with this for more than a day with this issue.

---

Added after 1 21 minutes:

Got the issue here, the functions decryptLoop() and joinContent() are to be static.
But again one more issue is that, I need to call a non-static function from inside decryptLoop(), how do I do it

[ejoshva]

I got the issue here. Issue is that the functions decryptLoop() and joinContents() are to be static.
But decrypteFile() is non-static function which I am calling from decreyptLoop(). How do I do it?
getting an error at this point like this.

[wysota]

Until you modify your decryption function to be re-entrant, it doesn't really matter how you call that function, it will return invalid results.

[ejoshva]

I have made it re-entrant and it's working now for QtConcurrent::mapped(). Thanks

But now I realise that what I am doing in two steps using mapped() can be done in single step using mappedReduced().

When I try to implement getting the below error again

error: no matching function for call to 'mappedReduced'
QFuture<QString> decryptedContent = QtConcurrent::mappedReduced(fileList,DecryptMap('A '),ReduceS());
^~~~~~~~~~~~~~~~~~~~~~~~~~~

struct DecryptMap {
    DecryptMap(const QChar &key) : m_key(key) {}
    typedef QString result_type;
    QString operator()(const QString &item)
    {
        Decrypt *decrypt = new Decrypt();
        qDebug()<<item;
        return decrypt->decryptFile(item,'A');
    }
    QChar m_key;
};
struct ReduceS
{
  typedef QString result_type;
  void operator()(QString & res, const QString & partial )
  {
    res.append(partial);
  }
};
QFuture<QString> decryptedContent =  QtConcurrent::mappedReduced(fileList,DecryptMap('A'),ReduceS());
 
QString Decrypt::decryptFile(QString sourceFile,QChar keystring)
{
     QString decryptedContent;
    decryptedContent.clear();
    string srcFile=sourceFile.toUtf8().constData();
    ifstream ffin;
 
    ffin.open(srcFile.c_str(),ios_base::in|ios_base::binary);
 
    char keyToEncryption= keystring.toLatin1();
    if(ffin.is_open())
    {
        char ch;
        while(!(ffin.read(&ch,1).eof()))
        {
            ch ^= keyToEncryption;
            decryptedContent.append(ch);
        }
    }
    ffin.close();
    return decryptedContent;
}

struct DecryptMap {
    DecryptMap(const QChar &key) : m_key(key) {}
    typedef QString result_type;
    QString operator()(const QString &item)
    {
        Decrypt *decrypt = new Decrypt();
        qDebug()<<item;
        return decrypt->decryptFile(item,'A');
    }
    QChar m_key;
};
struct ReduceS
{
  typedef QString result_type;
  void operator()(QString & res, const QString & partial )
  {
    res.append(partial);
  }
};
QFuture<QString> decryptedContent =  QtConcurrent::mappedReduced(fileList,DecryptMap('A'),ReduceS());

QString Decrypt::decryptFile(QString sourceFile,QChar keystring)
{
     QString decryptedContent;
    decryptedContent.clear();
    string srcFile=sourceFile.toUtf8().constData();
    ifstream ffin;

    ffin.open(srcFile.c_str(),ios_base::in|ios_base::binary);

    char keyToEncryption= keystring.toLatin1();
    if(ffin.is_open())
    {
        char ch;
        while(!(ffin.read(&ch,1).eof()))
        {
            ch ^= keyToEncryption;
            decryptedContent.append(ch);
        }
    }
    ffin.close();
    return decryptedContent;
}

To copy to clipboard, switch view to plain text mode 

[wysota]

Did you remember about proper includes?