QTableView memory consumption

[LordQt]

What do you mean???

When I parse a textfile with for example 150.000 lines the Applications memory usage grows up to 500-600 mb. When the tableview is filled it goes to 400 mb and still point there.

There must be a better solution for handling big textfiles, or not???

[muellerp]

What do you mean???

When I parse a textfile with for example 150.000 lines the Applications memory usage grows up to 500-600 mb. When the tableview is filled it goes to 400 mb and still point there.

There must be a better solution for handling big textfiles, or not???

QString is Unicode, so 2 bytes per character. It's not the lines, but the number of characters you have.

As you store the content of the file in the StandardItem you have it temporarily twice in memory.
After closing the file "only" once, all in QStandardItems.

I'm not surprised by your numbers.

The only way is to handle the file like a "database" - read only chunks until you find the needed part - and your model is then looking temporarily for each line into the file. But with a text file this is really a timing issue.

You may first read the file junk by junk and store it in a SQL database on disk, here the access times are faster. Still I don't think we will get wonders in performance.

[LordQt]

Thank you for yours suggestions ;O)

but I need to check content of the File before inserting the file into Database.

When I understand you right
I must insert the hole file in DB then visualize the content in QTreeView then reinserts the hole content to my real DB ... hmm

Thats inefficient isn´t it?

[muellerp]

As you don't want to start with a database, then you should do it similar like a database.

Databases don't keep the data itself in memory, just the index to the data on a harddrive.

In your case, you don't store the text itself, but an index to the position in the file. The index is then fast enough and not that memory consuming.

So I would first parse the file and at each position of a new line, you store the position in an index, so later you can do:

pos = index (row)
data = readData(file, pos, length)

Maybe you also store the length for faster processing.

Then your model takes for each line through this index the data when displaying from the harddisk.

I think this is then reasonable fast and the index takes much less memory.

[LordQt]

Is there any example out there .....

[LordQt]

QFile has no member that returns an index of a row.......????