Results 1 to 5 of 5

Thread: Storing and displaying "large" amount of data

  1. #1
    Join Date
    Jun 2013
    Posts
    20
    Thanks
    8
    Qt products
    Qt5
    Platforms
    Windows

    Default Storing and displaying "large" amount of data

    Hey guys!

    I have data stored in vectors that I would like to show in a QTextEdit. What I do now is I create a QTextDocument and a QTextCursor and I append all data as QStrings into that Document like this (altered some things so you can try to compile if needed):

    Qt Code:
    1. QTextDocument* parseDoc(std::vector<double>& time, std::vector<std::array<double,3>>& pos, std::vector<std::array<double,3>>& pos){
    2. parsedDocument = new QTextDocument();
    3. myDocumentCursor = new QTextCursor(parsedDocument);
    4.  
    5. QTextCharFormat textFormat = myDocumentCursor->charFormat();
    6. textFormat.setFontFamily("Courier");
    7. myDocumentCursor->setCharFormat(textFormat);
    8.  
    9.  
    10. QString formattedString;
    11. formattedString = QString("This is th Data of Some object. It was retrieved on %2\r\n").arg(QDateTime::currentDateTime().toString());
    12. myDocumentCursor->insertText(formattedString);
    13.  
    14. myDocumentCursor->insertBlock();
    15.  
    16. formattedString = QString("Epoch").leftJustified(21, ' ');
    17. formattedString.append(QString("Position x-Axis").leftJustified(21, ' '));
    18. formattedString.append(QString("Position y-Axis").leftJustified(21, ' '));
    19. formattedString.append(QString("Position z-Axis").leftJustified(21, ' '));
    20. formattedString.append(QString("Velocity x-Axis").leftJustified(21, ' '));
    21. formattedString.append(QString("Velocity y-Axis").leftJustified(21, ' '));
    22. formattedString.append(QString("Velocity z-Axis").leftJustified(16, ' '));
    23. myDocumentCursor->insertText(formattedString);
    24.  
    25. myDocumentCursor->insertBlock();
    26.  
    27. formattedString.clear();
    28.  
    29. for(int i = 0; i < time.size(); i++){
    30.  
    31. formattedString = QString("%1 %2 %3 %4 %5 %6 %7").arg(time[i], -16, 'f', -1)
    32. .arg(pos[i][0], -16, 'f', -1).arg(pos[i][1], -16, 'f', -1).arg(pos[i][2], -16, 'f', -1)
    33. .arg(vel[i][0], -16, 'f', -1).arg(vel[i][1], -16, 'f', -1).arg(vel[i][2], -16, 'f', -1);
    34.  
    35. myDocumentCursor->insertText(formattedString);
    36. myDocumentCursor->insertBlock();
    37. }
    38.  
    39. return parsedDocument;
    40.  
    41. }
    To copy to clipboard, switch view to plain text mode 

    Basically this works out quite smoothly, just takes a few minutes as I have roughly 500K lines of data. But there are 2 major problems that I ran into: 1. The memory usage after putting all data into the Document is around 1.2GB, 2. It takes a HUGE amount of time(~15min) to display the document in a QTextEdit (via QTextEdit::setDocument(QTextDocument*)).

    So obviously I am doing something wrong. So do you guys have an idea what I should be doing differently?

  2. #2
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Storing and displaying "large" amount of data

    Since you basically have a tabular structure, e.g. several values belonging to one record, each record having the same data fields, maybe consider using a table model and a table view?

    Cheers,
    _

  3. The following user says thank you to anda_skoa for this useful post:

    FreddyKay (27th November 2014)

  4. #3
    Join Date
    Jun 2013
    Posts
    20
    Thanks
    8
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: Storing and displaying "large" amount of data

    Quote Originally Posted by anda_skoa View Post
    Since you basically have a tabular structure, e.g. several values belonging to one record, each record having the same data fields, maybe consider using a table model and a table view?

    Cheers,
    _
    That might be possible, though I would rather like to have it in textform, as its way easier to use in my opinion. Is it normal, that QTextDocument is causing that much memory usage? If I put all those strings into a QStringList the amount of data in RAM is aprox. 75MB (just like a TextFile if I save all entries into a file). The TextDocument needs almost 15 times as much apparently. This makes me believe I made a mistake somewhere and there are maybe memory leaks or data dulpicates or somehting.

    Can you think of a way that would reduce memory usage and display time to more reasonable levels, where I can still use the QTextEdit / QPlainTextEdit?

  5. #4
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,230
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: Storing and displaying "large" amount of data

    I don't see any obvious memory leaks, unless you are calling this method more than once. Each time through you create a new QTextDocument instance; if you aren't getting rid of these and are calling this method over and over, then you'll keep accumulating dangling pointers to the old documents and using up more and more memory.

    This really does scream to be represented as a table. You're formatting it that way, so just use a QTableWidget or QTableView and turn off the grid lines.

    The amount of time could be because the document is recomputing its internal layout with each line you add to it and possibly pushing everything onto undo / redo stacks with each addition. The default value for the undoRedoEnabled property is true, and I don't see where you disable it. So you might have an undo stack with 500K blocks on it.
    Last edited by d_stranz; 27th November 2014 at 19:52.

  6. The following user says thank you to d_stranz for this useful post:

    FreddyKay (27th November 2014)

  7. #5
    Join Date
    Jun 2013
    Posts
    20
    Thanks
    8
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: Storing and displaying "large" amount of data

    Quote Originally Posted by d_stranz View Post
    I don't see any obvious memory leaks, unless you are calling this method more than once. Each time through you create a new QTextDocument instance; if you aren't getting rid of these and are calling this method over and over, then you'll keep accumulating dangling pointers to the old documents and using up more and more memory.

    This really does scream to be represented as a table. You're formatting it that way, so just use a QTableWidget or QTableView and turn off the grid lines.

    The amount of time could be because the document is recomputing its internal layout with each line you add to it and possibly pushing everything onto undo / redo stacks with each addition. The default value for the undoRedoEnabled property is true, and I don't see where you disable it. So you might have an undo stack with 500K blocks on it.
    Alright. Thanks a lot you two. I'll give QTableView a try, though I will also try disabling the undoRedo thingy.

Similar Threads

  1. Replies: 2
    Last Post: 3rd January 2012, 16:00
  2. Storing the program data in a "safe place"
    By kremuwa in forum Qt Programming
    Replies: 9
    Last Post: 5th August 2010, 11:23
  3. Replies: 1
    Last Post: 7th April 2010, 22:46
  4. Replies: 3
    Last Post: 25th August 2009, 14:03
  5. Guidance for displaying "updating" data!
    By Humble in forum Qt Programming
    Replies: 1
    Last Post: 3rd July 2007, 14:35

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.