Results 1 to 12 of 12

Thread: [PyQt] QTableView displying 100.000.000 rows

  1. #1
    Join Date
    Nov 2012
    Posts
    14
    Qt products
    Platforms
    Windows

    Default [PyQt] QTableView displying 100.000.000 rows

    Hi everyone,

    as the title says I'm trying to display 100.000.000 rows in my QTableView but I'm having some problems. When I launch my script, the widget seems to hang. After some investigations it turns out that the problem seems to be related to the headerData function which is called once for each row in the table. It does not seems to be normal, since I expect it to be called only for the currently displayed rows.
    Here a test code displaying the problem:

    Qt Code:
    1. import sys
    2. from PyQt4.QtCore import *
    3. from PyQt4.QtGui import *
    4.  
    5.  
    6. class MyWindow(QWidget):
    7. def __init__(self, *args):
    8. QWidget.__init__(self, *args)
    9.  
    10. # create table
    11. table = QTableView()
    12. tm = MyTableModel(self)
    13. table.setModel(tm)
    14.  
    15. # layout
    16. layout = QVBoxLayout()
    17. layout.addWidget(table)
    18. self.setLayout(layout)
    19.  
    20. class MyTableModel(QAbstractTableModel):
    21. def rowCount(self, parent):
    22. return 100000000
    23.  
    24. def columnCount(self, parent):
    25. return 10
    26.  
    27. def data(self, index, role):
    28. if not index.isValid():
    29. return QVariant()
    30. elif role != Qt.DisplayRole:
    31. return None
    32. return str((index.row(), index.column()))
    33.  
    34. def headerData(self, section, orientation, role):
    35. print "headerData", section
    36. if role == Qt.DisplayRole:
    37. return section
    38. return None
    39.  
    40. if __name__ == "__main__":
    41. app = QApplication(sys.argv)
    42. w = MyWindow()
    43. w.resize(900, 600)
    44. w.show()
    45. sys.exit(app.exec_())
    To copy to clipboard, switch view to plain text mode 

    Anyone can point me out what I'm doing wrong? is it a bug? The test has been run on windows 7 (64 bit), PyQt 4.11.
    Thanks in advance!

  2. #2
    Join Date
    Dec 2009
    Location
    New Orleans, Louisiana
    Posts
    791
    Thanks
    13
    Thanked 153 Times in 150 Posts
    Qt products
    Qt5
    Platforms
    MacOS X

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    Does a user interface with 100 million rows really sound like a good idea?

  3. #3
    Join Date
    Nov 2012
    Posts
    14
    Qt products
    Platforms
    Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    In the years of big data the answer is probably yes

  4. #4
    Join Date
    Dec 2009
    Location
    New Orleans, Louisiana
    Posts
    791
    Thanks
    13
    Thanked 153 Times in 150 Posts
    Qt products
    Qt5
    Platforms
    MacOS X

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    Quote Originally Posted by Gad82 View Post
    In the years of big data the answer is probably yes
    Good luck with that then...

    Edit: Regarding your original question, are you by chance using QHeaderView::ResizeToContents?
    Last edited by jefftee; 15th June 2015 at 17:42.

  5. #5
    Join Date
    Nov 2012
    Posts
    14
    Qt products
    Platforms
    Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    No, as shown in the test script I only set up a simple model and then put it into the QTableView....

  6. #6
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,229
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    In the years of big data the answer is probably yes
    In the years of big data, no one actually looks at the big data. If they look at it at all, they look at visualizations that condense the data into something a human viewer can comprehend. The individual items are for the most part, nearly irrelevant. No one, absolutely no one, will look at anything more than the first page or two of a table with 100 million entries, so why even bother with such a thing?

    If you feel you must do this, then you should implement a model which retrieves only what the user is currently viewing on the screen and not attempt to build either a model or a view that contains all of it.

  7. #7
    Join Date
    Dec 2009
    Location
    New Orleans, Louisiana
    Posts
    791
    Thanks
    13
    Thanked 153 Times in 150 Posts
    Qt products
    Qt5
    Platforms
    MacOS X

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    Looks like you are not differentiating between horizontal and vertical headers. If you don't want header data for rows, then you should be returning QVariant() when the orientation is not horizontal.

    I don't code in Python, if None is not the same as QVariant(), then you should change your code to return QVariant() instead of None. Just guessing but perhaps your headerData is being repeatedly called because you're not returning QVariant() in the case where you have no data to return.

  8. #8
    Join Date
    Apr 2019
    Posts
    1
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    Even when returning QVariant() in header data the QTableView header still tries to allocate a vector the size of the row count.
    Is it possible to have hundred millions of rows in the model without an out of memory crash in the table?

  9. #9
    Join Date
    May 2020
    Posts
    1
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    I hit the same problem and when I looked into it, the problem is QHeaderView. It internally calculates the offset of each row in pixels but it uses an int so limited to 2 billion and if you overflow it you end up negative. With 100M rows if the rows are over 20 pixels or so, which on a 4K monitor in particular is very likely, then these pixel offsets overflow and the code goes haywire causing very long loops that make it appear to have hung.

    I wish there was a QAbstractHeaderView that things like QTableView used with QHeaderView just an implementation we could replace with something better. But when I looked into the code QHeaderView has a very intimate knowledge and tie into the private data of other views and models so there seems to be very poor separation of concerns in the code and it would be very difficult to replace it.

    Off to look for some other way to show 100M rows!

  10. #10
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,229
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    Instead of making a model that reports it has 100M rows (which is what the header is relying on in its calculations), devise a model that can page through a subset of your data and report fewer rows so the header view stays well-behaved.

    Thinking that your table must accommodate all 100M rows is an example of "confusing the map for the territory". Your underlying data structure (the territory) might contain 100M entries, but your view into it (the map) does not have to.

    One possible solution is to implement a sliding window onto your underlying data that contains say, twice as many rows as can be viewed onscreen. When you scroll up or down, you slide the window along, moving the first and last index rows accordingly. This should also significantly improve your UI response time, since the view s only have to base their calculations on a few hundred rows instead of thousands or millions. This could probably be done with a proxy model. If having an accurate row number on the vertical header is important, you can re-implement your model's headerData() method to return a string containing the actual row number from the underlying data.
    Last edited by d_stranz; 17th May 2020 at 17:49.
    <=== The Great Pumpkin says ===>
    Please use CODE tags when posting source code so it is more readable. Click "Go Advanced" and then the "#" icon to insert the tags. Paste your code between them.

  11. #11
    Join Date
    Sep 2019
    Posts
    4
    Thanks
    1
    Qt products
    Qt5
    Platforms
    Unix/X11 Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    I thought the default QTableView already did something like this?

  12. #12
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,229
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: [PyQt] QTableView displying 100.000.000 rows

    I thought the default QTableView already did something like this?
    It does, as far as displaying rows is concerned. But apparently the header view layout takes the entire model into account when calculating row location, at least according to @bpepers' post. I am guessing that this is how it determines whether a row is visible on screen or not.
    <=== The Great Pumpkin says ===>
    Please use CODE tags when posting source code so it is more readable. Click "Go Advanced" and then the "#" icon to insert the tags. Paste your code between them.

Similar Threads

  1. PyQt - QTableView with comboBox
    By elanari in forum Qt Programming
    Replies: 1
    Last Post: 9th May 2011, 15:13
  2. QTableView with icons in rows
    By kasper360 in forum Newbie
    Replies: 6
    Last Post: 5th April 2011, 07:24
  3. Disable row/rows in QTableView
    By scott_hollen in forum Qt Programming
    Replies: 6
    Last Post: 28th March 2011, 19:16
  4. QTableView 2 rows inside every row
    By skuda in forum Qt Programming
    Replies: 3
    Last Post: 22nd April 2009, 10:23
  5. Copying QTableView rows
    By derrickbj in forum Qt Programming
    Replies: 1
    Last Post: 28th September 2006, 01:00

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.