Big Problem search through huge amount of data ca 30 000 strings takes to long

[Ini]

Hello,

this is my Code:

void SomeClass::searchIn_identifier_listWidget()
{
    QList<Identifier*> tempList;
 
    QString text0 = ui->identifier_lineEdit_search_0->text();
    QString text1 = ui->identifier_lineEdit_search_1->text();
    QString text2 = ui->identifier_lineEdit_search_2->text();
    QString text3 = ui->identifier_lineEdit_search_3->text();
    QString text4 = ui->identifier_lineEdit_search_4->text();
 
    for (int i = 0; i < Storage::identifier.count(); i++) {
        if (Storage::identifier.at(i)->dataString.contains(text0,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text1,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text2,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text3,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text4,Qt::CaseInsensitive))
        {
            tempList << Storage::identifier.at(i);
        }
    }
 
    ui->identifier_listWidget->clear();
    for (int i = 0; i < tempList.count(); i++) {
        QTest::qWait(1);
        listWidget_addItemPlusTooltip(ui->identifier_listWidget,tempList.at(i)->dataString);
    }
}

void SomeClass::searchIn_identifier_listWidget()
{
    QList<Identifier*> tempList;

    QString text0 = ui->identifier_lineEdit_search_0->text();
    QString text1 = ui->identifier_lineEdit_search_1->text();
    QString text2 = ui->identifier_lineEdit_search_2->text();
    QString text3 = ui->identifier_lineEdit_search_3->text();
    QString text4 = ui->identifier_lineEdit_search_4->text();

    for (int i = 0; i < Storage::identifier.count(); i++) {
        if (Storage::identifier.at(i)->dataString.contains(text0,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text1,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text2,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text3,Qt::CaseInsensitive) &&
            Storage::identifier.at(i)->dataString.contains(text4,Qt::CaseInsensitive))
        {
            tempList << Storage::identifier.at(i);
        }
    }

    ui->identifier_listWidget->clear();
    for (int i = 0; i < tempList.count(); i++) {
        QTest::qWait(1);
        listWidget_addItemPlusTooltip(ui->identifier_listWidget,tempList.at(i)->dataString);
    }
}

To copy to clipboard, switch view to plain text mode 

So as you can see i want to search though someting in this case a QList and load the Strings into the ListWidget which match the search. The algorythm should stay like this I think. It takes like minutes until the Widget is filled with the new data. i have like 30 000 strings in this List. And all GUI should stay responsive to the user thats why the qWait is there. Maybe thats why it is so slow, but how can the GUI stay responsive without qwait?

What can I do???

Can somebody help me?

edit: Without qwait the whole GUI in ther widget where this QListWidget is is locked. And it also takes like 1 minute to process all data into the QListWidget

thx

[Lesiok]

First optimization - five times less method Storage::identifier.at(i) calls :

QString temp_str;
for (int i = 0; i < Storage::identifier.count(); i++) {
        temp_str = Storage::identifier.at(i)->dataString;
        if (temp_str.contains(text0,Qt::CaseInsensitive) &&
            temp_str.contains(text1,Qt::CaseInsensitive) &&
            temp_str.contains(text2,Qt::CaseInsensitive) &&
            temp_str.contains(text3,Qt::CaseInsensitive) &&
            temp_str.contains(text4,Qt::CaseInsensitive))
        {
            tempList << Storage::identifier.at(i);
        }
    }

QString temp_str;
for (int i = 0; i < Storage::identifier.count(); i++) {
        temp_str = Storage::identifier.at(i)->dataString;
        if (temp_str.contains(text0,Qt::CaseInsensitive) &&
            temp_str.contains(text1,Qt::CaseInsensitive) &&
            temp_str.contains(text2,Qt::CaseInsensitive) &&
            temp_str.contains(text3,Qt::CaseInsensitive) &&
            temp_str.contains(text4,Qt::CaseInsensitive))
        {
            tempList << Storage::identifier.at(i);
        }
    }

To copy to clipboard, switch view to plain text mode 

[anda_skoa]

Also, since QTest::qWait() suggests this is a unit test, is it really that problematic?

In a real application blocking the UI would be bad, but in an automated test you will have to wait for the result state anyway.

Cheers,
_

[Ini]

Can qWait not stay in the endprogramm?

I cannot cut qWait cause GUI stays unresponsive then. I would consider this as big problem...

---

Added after 20 minutes:

First optimization - five times less method Storage::identifier.at(i) calls :

QString temp_str;
for (int i = 0; i < Storage::identifier.count(); i++) {
        temp_str = Storage::identifier.at(i)->dataString;
        if (temp_str.contains(text0,Qt::CaseInsensitive) &&
            temp_str.contains(text1,Qt::CaseInsensitive) &&
            temp_str.contains(text2,Qt::CaseInsensitive) &&
            temp_str.contains(text3,Qt::CaseInsensitive) &&
            temp_str.contains(text4,Qt::CaseInsensitive))
        {
            tempList << Storage::identifier.at(i);
        }
    }

QString temp_str;
for (int i = 0; i < Storage::identifier.count(); i++) {
        temp_str = Storage::identifier.at(i)->dataString;
        if (temp_str.contains(text0,Qt::CaseInsensitive) &&
            temp_str.contains(text1,Qt::CaseInsensitive) &&
            temp_str.contains(text2,Qt::CaseInsensitive) &&
            temp_str.contains(text3,Qt::CaseInsensitive) &&
            temp_str.contains(text4,Qt::CaseInsensitive))
        {
            tempList << Storage::identifier.at(i);
        }
    }

To copy to clipboard, switch view to plain text mode 

is this really faster? whats the problem with .at(i) calls

[Lesiok]

---

Added after 20 minutes:

is this really faster? whats the problem with .at(i) calls

The problem is "this is a procedure calling". What do you think is faster: one call or five calls ?
Besides, my experience is that many operations on the Qt containers is very slow in debug mode because of the control parameters with ASSERT.

PS.

Have you tried to determine which loop will give you such a long time?

[Ini]

I dont know im not expert in memeory management. In my opionion both "point" to a location. I will check on that.

edit: I work in Release Mode.

But thats really not the Problem. Problem is the second loop.

I added qApp->processEvents. That helps but then there is a another Problem.
This Slot is connected to textChanged() of the LineEdits. So when text changes the slot will get triggerd, then text changes again, and it seems the slot then stops and performs the new instance of the slotcall and when this new instance is finsihed the old one continues. Am I right there? How to solve that upcoming problem?

[anda_skoa]

Can qWait not stay in the endprogramm?

If you want the real program to link to QtTest, sure.
Usually one doesn't want artifical dependencies.

I cannot cut qWait cause GUI stays unresponsive then. I would consider this as big problem...

Then I would suggest looking for a solution instead.

E.g. putting the result list into a model and having a QListView work on that model.

Cheers,
_

[Ini]

If you want the real program to link to QtTest, sure.
Usually one doesn't want artifical dependencies.


Then I would suggest looking for a solution instead.

E.g. putting the result list into a model and having a QListView work on that model.

Cheers,
_

why is that faster then QListWidget

"Usually one doesn't want artifical dependencies."

I dont find a good description what that is. Can you explain pls?

[anda_skoa]

I dont know im not expert in memeory management. In my opionion both "point" to a location.

Yes, but one is a direct "pointer" to the data, the other is repeatedly calculating the "pointer".
Using a variable can even more likely lead to the pointer to be cached by the CPU than the result of a function evaluation which just incidentally (as far as the CPU knowns) returns the same value five times in a row.

I added qApp->processEvents. That helps but then there is a another Problem.

Yes, this new problem is called "nested event loop".

This Slot is connected to textChanged() of the LineEdits. So when text changes the slot will get triggerd, then text changes again, and it seems the slot then stops and performs the new instance of the slotcall and when this new instance is finsihed the old one continues.

Yes, nested event loops can lead to behavior similar to re-entrancy.

why is that faster then QListWidget

Because you don't have to create all items, a list view only displays as many entries as it can fit.

I dont find a good description what that is. Can you explain pls?

A dependency on a library that you actually don't need.

Cheers,
_