Results 1 to 8 of 8

Thread: Need lazy load approach for treeview & wanted to maintain only viewing items in mem

  1. #1
    Join Date
    Jun 2015
    Location
    India
    Posts
    185
    Thanks
    8
    Thanked 14 Times in 14 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Need lazy load approach for treeview & wanted to maintain only viewing items in mem

    Hi,

    I wanted to build a tree with lazy loading (not lazy population), The problem I have is, I have lots of data (around 10 billion to 100 billion rows) building a tree with this much data is obviously crashing tree as (memory issue). So I wanted make lazy loading (At a time I want to maintain those items which I show in view).

    I have few doubts with this approach

    1. I can use canFetchMore() & fetchMore() functions. when I scroll down I want to get the below items (which is possible with this functions) and I want to delete items which gone out of view.
    Loading data when scroll down can get from above functions but how to get the data when I scroll up ??

    2. how can I delete items which gone out of view when I scroll ??

    3. how can I maintain scroll bar size with this approach ??

    4. last but one is I wanted to implement filter also on top of this.

    I have seen fetchMore() example but its for list & where items are not getting deleted when gone out of view. I want to get same behavior but with a tree & want to delete items which gone out of view & want to maintain scroll bar properly.

    any suggestions please.
    Thanks :-)

  2. #2
    Join Date
    Oct 2009
    Location
    Germany
    Posts
    120
    Thanked 42 Times in 41 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Need lazy load approach for treeview & wanted to maintain only viewing items in

    Hello,

    I ran into some problems when displaying more than 72 million rows in a table (the view only showed garbage). The reason here was that internally the view components do all computations with integer (signed 32 bit) and the default row height is 30, so the view produces integer overflows when working with that amount of data. There is already a bug note on this in the Qt bug tracker.

    For working around this I have chosen the following approach (maybe this could help you too):
    1. I divided my model into "pages" consisting of up to 60 million rows. The view will display only the currently loaded "page" and the scrollbar is used for scrolling inside this "page".
    2. I added a - and + button at the top and bottom of the view vertical scrollbar. Clicking the + button will move the model page by half page size down and pressing - button will move it by half page size up. The view will get updated when clicking those buttons.

    In your situation you can unload old / load new data into the model when the +/- buttons are clicked.

    Regards
    ars

  3. #3
    Join Date
    Jun 2015
    Location
    India
    Posts
    185
    Thanks
    8
    Thanked 14 Times in 14 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Need lazy load approach for treeview & wanted to maintain only viewing items in

    Quote Originally Posted by ars View Post
    Hello,

    I ran into some problems when displaying more than 72 million rows in a table (the view only showed garbage). The reason here was that internally the view components do all computations with integer (signed 32 bit) and the default row height is 30, so the view produces integer overflows when working with that amount of data. There is already a bug note on this in the Qt bug tracker.
    Hi thanks,
    you mean, can't we show/handle more than 70+ million items in view ?

    Regards,
    Durga.
    Thanks :-)

  4. #4
    Join Date
    Oct 2009
    Location
    Germany
    Posts
    120
    Thanked 42 Times in 41 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Need lazy load approach for treeview & wanted to maintain only viewing items in

    Hello,

    have a look at https://bugreports.qt.io/browse/QTBUG-28631 describing the bug in Qt 4.7 and 4.8. They have not fixed it.

    I also tried working around this limitation by using a tree view instead of of a table view and group my data into 60 million sub rows that get expanded or collapsed by the user clicking on the parent item. Here the parent item represented a range of rows. This was so incredibly slow that it was unusable.

    Another point is how to use the scrollbar when navigating through 10 billion rows. Let's assume the scrollbar covers 1000 pixels on the screen, then each scrollbar position maps to 10 million rows of data. For navigating through such big data sets, I think a scrollbar alone is not adequate. This also applies to my application. I have a graphical representation of the data (trace) where the user can put a marker at some point that looks interesting and then may scroll the table to the position of the marker to see the raw data in detail.

    Hope that helps you a little bit.

    Regards
    ars

  5. #5
    Join Date
    Jun 2015
    Location
    India
    Posts
    185
    Thanks
    8
    Thanked 14 Times in 14 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Need lazy load approach for treeview & wanted to maintain only viewing items in

    Thanks, I will look into the link.

    Edited : gone through the link, Looks like they have not fixed the bug instead they suggested to build 64-bit application, my application is 64 bit any how .

    BTW: how did you do memory management in this case, As you have huge row & we need to build tree before population it onto view.
    So did you manage the memory. did you create entire tree (items) and populate it or in some other way ??
    Last edited by prasad_N; 4th July 2015 at 22:59.
    Thanks :-)

  6. #6
    Join Date
    Oct 2009
    Location
    Germany
    Posts
    120
    Thanked 42 Times in 41 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Need lazy load approach for treeview & wanted to maintain only viewing items in

    I have 2 different applications for this. In one application the data are stored in a file, all with equal record size. So if I know a row number, I can compute the location in the file and load the data from there. Of course I do some caching of data to improve performance.

    In the second application a single column of data is stored in memory. From the context of the application I know that there is a limitation of 200 million data points (each a double) and I have to keep those data in memory (simple std::vector) anyway because a bunch of numerical algorithms are applied to the data basically reducing the millions of double values into pass/fail results. So I need up to 1.6GB of ram. Allocating this is no big problem in today PCs with 64 bit OS and 8 or 16GB ram. That's for the first column in my table. There are further columns, but the elements of these columns are computed on the fly from the data in the first column when the data in the other column gets read by a view. Therefore I do not need to allocate memory for these columns.

    As I mentioned before, I store my data in a std::vector object. The model used by my table view does not store data by itself. Instead, it gets a reference to my data vector and accesses the vector for reading the data. That prevents duplicating data just for displaying them in a table. So in my case, the Qt table model serves as a "glue" between my data (std::vector) and the Qt table view objects.

    Best regards
    ars

  7. #7
    Join Date
    Jun 2015
    Location
    India
    Posts
    185
    Thanks
    8
    Thanked 14 Times in 14 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Need lazy load approach for treeview & wanted to maintain only viewing items in

    Oops.. Thanks for the detailed answer.
    In my case this may not work because before populating a tree, need to build the tree this is not a case with table or list view. This is the reason I wanted to go with lazy population model and left with the above doubts.
    Thanks :-)

  8. #8
    Join Date
    Jun 2015
    Location
    India
    Posts
    185
    Thanks
    8
    Thanked 14 Times in 14 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Need lazy load approach for treeview & wanted to maintain only viewing items in

    Any suggestions from others ???
    Thanks :-)

Similar Threads

  1. How many items can Treeview /Tableview holds ?
    By prasad_N in forum Qt Programming
    Replies: 4
    Last Post: 6th July 2015, 06:32
  2. Nesting TableViews under TreeView items
    By marcuso in forum Qt Programming
    Replies: 9
    Last Post: 24th March 2012, 19:00
  3. Replies: 1
    Last Post: 4th August 2011, 23:36
  4. Replies: 1
    Last Post: 9th August 2009, 20:51
  5. speed of setdata - lots of items in treeview
    By Big Duck in forum Qt Programming
    Replies: 4
    Last Post: 6th July 2006, 13:53

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.