Results 1 to 20 of 21

Thread: beginRemove() endRemove = noRemove

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Join Date
    Nov 2008
    Posts
    183
    Thanks
    13
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default beginRemove() endRemove = noRemove

    I have a QTableView and a model based on QAbstractTableModel. The model is periscoping over what will be a massive dataset. Naturally we had to have that *(&^)(*&^(*&ing flick scrolling on the view so we had to have an external scrollbar to cover the entire database range. (I thought the user should be punished and made to flick 80 rows at a time through a multi-million row set, but that is another story.)

    Quite dutifully my model calls beginRemove(), deletes data, then calls endRemove(). Scrolling down stutters and jerks like I expect it would. Scrolling up is smooth as silk. I guess I should have made them fill out that dataset to maximum size for initial testing because trying to fit north of 1TB binary data into 4G of RAM in text form would have pointed out the issue early on. The data is removed from the model, but not the view.

    I will dig further into the QTableView documentation after supper, but here is the question: Does the default implementation of QTableView ignore beginRemove() and endRemove()? It appears to ignore this but respect beginAppend() and endAppend().

    Thanks,

    Oh. Qt 4.8.x on Linux Target is embedded, but problem visible on both.

  2. #2
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: beginRemove() endRemove = noRemove

    Quote Originally Posted by RolandHughes View Post
    Does the default implementation of QTableView ignore beginRemove() and endRemove()?
    No, it doesn't. beginRemove*() and endRemove*() cause persistent indexes to be updated and then emit respective (rows|columns)Removed() signals.

    wysota@Teletraan3:~/kompilacje/qt-everywhere-opensource-src-4.8.6/src/gui/itemviews$ grep rowsRemoved qtableview.cpp
    disconnect(d->model, SIGNAL(rowsRemoved(QModelIndex,int,int)),
    connect(model, SIGNAL(rowsRemoved(QModelIndex,int,int)),
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  3. #3
    Join Date
    Nov 2008
    Posts
    183
    Thanks
    13
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: beginRemove() endRemove = noRemove

    Thanks for taking the time to respond. Not exactly an answer, but a response, so thanks for taking the time to do it. I looked at the source file you indicated and it confirms what I suspected/see happening. The "view", meaning the scrolling region, is not getting content trimmed/updated. The model is physically removing data from itself and some black bag stuff is being updated behind the scene, but the "text" the scrolling area knows about is not being cleaned. This appears to be organized for user drive deletions not model driven.

    Before we go off on a tangent allow me to provide a picture. Other than being illustrative and conceptually what is happening, it has no relationship to what is actually being done.

    Consider the model to be a bookshelf of some fixed size able to support limitless weight but only so many feet wide. When the db scrollbar (scrolling by ID, not by pixel like the view scroll area) is moved it sees if the model has any rows for the new ID and then tells the view to scroll N or -N pixels to get to the first of what could be many rows for that ID (i.e. Some books are many inches thicker than others). When the ID is not there the it tells the model to get more books. The model shoves those books onto one end or the other of the shelf. It then makes note how many books need to fall off the other end for things to fit, does a beginRemove(), deletes the data, then does an endRemove(). It's not a math problem with pixels/row because it does scroll to correct information.

    Debug statements for the hidden viewport scroll bar show the pixel position growing and growing and growing. Despite the fact the model has physically removed the data and done the begin/end remove stuff, the scroll area is not tidied up.

    Oh well, long day. Some wine, some sleep, and tomorrow add a throw away user delete method to verify if this was setup to be user driven instead of data driven and needs selected row(s).

    Thanks again.

  4. #4
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,230
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: beginRemove() endRemove = noRemove

    Oh well, long day. Some wine, some sleep.
    My preferred solution to vexing programming problems. I usually insert "some good jazz" between the wine and sleep, but often the wine and jazz are concurrent.

  5. #5
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: beginRemove() endRemove = noRemove

    Quote Originally Posted by RolandHughes View Post
    Thanks for taking the time to respond. Not exactly an answer, but a response, so thanks for taking the time to do it.
    Not very kind of you to say that.

    I looked at the source file you indicated and it confirms what I suspected/see happening. The "view", meaning the scrolling region, is not getting content trimmed/updated. The model is physically removing data from itself and some black bag stuff is being updated behind the scene, but the "text" the scrolling area knows about is not being cleaned.
    What exactly do you mean by "the text the scrolling area knows about"? The view does not hold the model data anywhere, that's the model's responsibility.

    Before we go off on a tangent allow me to provide a picture. Other than being illustrative and conceptually what is happening, it has no relationship to what is actually being done.

    Consider the model to be a bookshelf of some fixed size able to support limitless weight but only so many feet wide. When the db scrollbar (scrolling by ID, not by pixel like the view scroll area) is moved it sees if the model has any rows for the new ID and then tells the view to scroll N or -N pixels to get to the first of what could be many rows for that ID (i.e. Some books are many inches thicker than others). When the ID is not there the it tells the model to get more books. The model shoves those books onto one end or the other of the shelf. It then makes note how many books need to fall off the other end for things to fit, does a beginRemove(), deletes the data, then does an endRemove(). It's not a math problem with pixels/row because it does scroll to correct information.

    Debug statements for the hidden viewport scroll bar show the pixel position growing and growing and growing. Despite the fact the model has physically removed the data and done the begin/end remove stuff, the scroll area is not tidied up.
    I don't know how you are implementing your model and your view but according to me you have two choices compliant with the item-views architecture. Basically you can either report the proper size of the model and cache its data partially or modify the model to fit what is displayed in the view using insert/remove rows. I'm assuming you have taken the latter approach so I will focus on the former one first.

    Here is an example which implements a model with 40M items where only 1k items are kept in memory at once (unfortunately the index is int-based so we can't have much more rows).

    Qt Code:
    1. #include <QApplication>
    2. #include <QAbstractTableModel>
    3. #include <QTableView>
    4. #include <QContiguousCache>
    5. #include <QtDebug>
    6.  
    7. class Model : public QAbstractTableModel {
    8. public:
    9. Model(int from, int to, int cacheSize, QObject *parent = 0) : QAbstractTableModel(parent),
    10. m_from(from),
    11. m_to(to)
    12. {
    13. m_cache.setCapacity(cacheSize);
    14. }
    15. int rowCount(const QModelIndex &parent = QModelIndex()) const { return parent.isValid() ? 0 : m_to-m_from+1; }
    16. int columnCount(const QModelIndex &parent = QModelIndex()) const { return 1; }
    17. QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const {
    18. if(role != Qt::DisplayRole) return QVariant();
    19. int row = index.row();
    20. if(row > m_to) return QVariant();
    21. if(row < m_from) return QVariant();
    22. while(row > m_cache.lastIndex())
    23. m_cache.append(populate(m_cache.lastIndex()+1));
    24. while(row < m_cache.firstIndex())
    25. m_cache.prepend(populate(m_cache.firstIndex()-1));
    26. return m_cache.at(row);
    27. }
    28. protected:
    29. int populate(int row) const {
    30. return row*2; // simple populate function
    31. }
    32.  
    33. private:
    34. mutable QContiguousCache<int> m_cache;
    35. int m_from;
    36. int m_to;
    37. };
    38.  
    39. int main(int argc, char *argv[])
    40. {
    41. QApplication a(argc, argv);
    42. QTableView view;
    43. Model model(0, 40000000L, 1000);
    44. view.setModel(&model);
    45. view.show();
    46. return a.exec();
    47. }
    To copy to clipboard, switch view to plain text mode 

    You can fine tune the model by providing some look-ahead buffer, add data in batches and/or use a thread to populate the look-ahead buffer. The docs for QContignousCache might give you some ideas.

    As for the other approach, maybe we could have a look at your code or some example which behaves in a similar fashion? Are you making use of canFetchMore()/fetchMore()?
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  6. #6
    Join Date
    Nov 2008
    Posts
    183
    Thanks
    13
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: beginRemove() endRemove = noRemove

    Quote Originally Posted by d_stranz View Post
    My preferred solution to vexing programming problems. I usually insert "some good jazz" between the wine and sleep, but often the wine and jazz are concurrent.
    I've been more into "space music" lately, especially while coding.


    Added after 51 minutes:


    What exactly do you mean by "the text the scrolling area knows about"? The view does not hold the model data anywhere, that's the model's responsibility.
    What I mean is the model physically has the rows removed and has called the beingRemove/endRemove stuff accordingly but the pixel count for the scrolling area increases with each append without ever decreasing for the remove.


    I don't know how you are implementing your model and your view but according to me you have two choices compliant with the item-views architecture. Basically you can either report the proper size of the model and cache its data partially or modify the model to fit what is displayed in the view using insert/remove rows. I'm assuming you have taken the latter approach so I will focus on the former one first.
    There-in lies the rub. One database record could be one row in the table or it could be 100+ depending on selection criterea. The proprietary data encoding cannot be counted or expanded with SQL. In short the rowCount of the universe cannot realistically be known. I was not using a contiguous cache but rather a QList of QLists since I have x-y columns depending on configuration. My model communicates with another part of the system which pulls chunks of data out of the database. My model then expands it based on the settings creating a QList of column data which gets added to the QList of QLists for rows. The rowCount returned is always the current size of my row QList. The size of that QList is checked after each manipulation and entries from the other end are removed until we are below the target.

    As for the other approach, maybe we could have a look at your code or some example which behaves in a similar fashion? Are you making use of canFetchMore()/fetchMore()?
    Not at this time. I experimented with that several months ago when I first did this table but it didn't seem viable. It is all well and good for adding chunks, but not so good for the periscope model.

    No, I cannot change the database. I fought that battle and have all of the changes I will ever get.

    I will spend the rest of the day working on it. If I don't get it then I will have to spend half a day coding something completley unrelated from scratch which demonstrates the problem without violating NDA, SECRETS, or any other agreement.

    Thanks again.
    Last edited by RolandHughes; 16th December 2014 at 12:31.

  7. #7
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: beginRemove() endRemove = noRemove

    Quote Originally Posted by RolandHughes View Post
    What I mean is the model physically has the rows removed and has called the beingRemove/endRemove stuff accordingly but the pixel count for the scrolling area increases with each append without ever decreasing for the remove.
    The scroll is purely virtual, QTableView inherits QAbstractScrollArea where no actual scrolling is really done. What you draw is simply shifted based on the values of the scrollbars (ranging from verticalScrollBar()->minimum() to verticalScrollBar()->maximum()).

    Qt Code:
    1. #include <QApplication>
    2. #include <QTableView>
    3. #include <QStandardItemModel>
    4. #include <QtDebug>
    5. #include <QScrollBar>
    6.  
    7. int main(int argc, char **argv) {
    8. QApplication app(argc, argv);
    9. QTableView view;
    10. QStandardItemModel model(400,1);
    11. view.setModel(&model);
    12. view.setVerticalScrollMode(QAbstractItemView::ScrollPerPixel);
    13. view.show();
    14. app.processEvents();
    15. qDebug() << view.verticalScrollBar()->minimum() << view.verticalScrollBar()->maximum();
    16. model.removeRows(0, 50);
    17. app.processEvents();
    18. qDebug() << view.verticalScrollBar()->minimum() << view.verticalScrollBar()->maximum();
    19. return 0;
    20. }
    To copy to clipboard, switch view to plain text mode 

    yields:

    0 11831
    0 10331

    I do not observe any pixel count not decreasing upon removal.
    There-in lies the rub. One database record could be one row in the table or it could be 100+ depending on selection criterea. The proprietary data encoding cannot be counted or expanded with SQL. In short the rowCount of the universe cannot realistically be known.
    That is not a problem. You can always add new rows to the model. Removing seems to be your problem which the use of QContignousCache (or any other similar approach) avoids in an elegant way. And in my opinion you should really look into using canFetchMore()/fetchMore() (if your backend API allows to implement it).
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  8. #8
    Join Date
    Nov 2008
    Posts
    183
    Thanks
    13
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: beginRemove() endRemove = noRemove

    I'm working on a stand alone example which removes all database, etc and doesn't expose anything requiring a wink and a nod to a blind man to know.

    What is getting lost in translation here is the "book" concept. An entire book has to fall off, not just a few pages.

    The model has to keep track of which database IDs (hmm Dewey Decimal Numbers actually work well for this). What is set is the number of database IDs we allow the model to utilize at any given time. The number of rows is, however, many it needs to be. Each database ID could be dozens or hundreds of rows in the table. When an ID gets shoved out of the list all of the rows for that ID get nuked from the model. There is no realistic method of retrieving and reconstructing part of a book, has to be the entire thing, you don't get to rip the first chapter out and stuff it back in when you are done.

    It's not of my choosing, it simply is what it is.

    Btw, after others punted on this and it was handed to me I changed the model to be based on QAbstractTableModel which may not have been the wisest choice.

    Now that I'm typing this and thinking about it, what I really need is a model of a model. One model containing database IDs which feeds/communicates with the next model which does the expansion and removal.

  9. #9
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: beginRemove() endRemove = noRemove

    Quote Originally Posted by RolandHughes View Post
    I'm working on a stand alone example which removes all database, etc and doesn't expose anything requiring a wink and a nod to a blind man to know.

    What is getting lost in translation here is the "book" concept. An entire book has to fall off, not just a few pages.

    The model has to keep track of which database IDs (hmm Dewey Decimal Numbers actually work well for this). What is set is the number of database IDs we allow the model to utilize at any given time. The number of rows is, however, many it needs to be. Each database ID could be dozens or hundreds of rows in the table. When an ID gets shoved out of the list all of the rows for that ID get nuked from the model. There is no realistic method of retrieving and reconstructing part of a book, has to be the entire thing, you don't get to rip the first chapter out and stuff it back in when you are done.
    I don't understand how that is relevant to anything. In the end it all boils down to adding and removing rows. The internal representation doesn't really matter here as long as you know which row is which. If your model is hierarchical (you have IDs and each ID expands to a number of rows) then maybe you should simply have a hierarchical model (derived from QAbstractItemModel with parents and stuff) and not a flat one? Of course you will not be able to use QTableView to view it properly unless you employ a proxy model that will flatten your hierarchical model.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


  10. #10
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,230
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: beginRemove() endRemove = noRemove

    Of course you will not be able to use QTableView to view it properly unless you employ a proxy model that will flatten your hierarchical model.
    And this is doable and has been in KDE's FlatProxyModel, which I found after a lot of trial, error, more error, and searching.

    By simply adding / removing top-level IDs from the tree model, the flat model will automatically add / remove all the rows corresponding to the IDs. Your "list of lists" model should be easily mappable to a tree model.

    Flick away!
    Last edited by d_stranz; 17th December 2014 at 00:37.

  11. #11
    Join Date
    Nov 2008
    Posts
    183
    Thanks
    13
    Thanked 2 Times in 2 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11

    Default Re: beginRemove() endRemove = noRemove

    Thanks d_stranz. You wouldn't happen to know if that works on embedded Linux under -qws woudl you? I suspect it relies quite a bit on KDE.

    Sadly I have not been able to put together an example which can be approved for external consumption so I'm on my own.

  12. #12
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: beginRemove() endRemove = noRemove

    Quote Originally Posted by RolandHughes View Post
    I suspect it relies quite a bit on KDE.
    It's a purely Qt class, you can see the includes here: http://api.kde.org/bundled-apps-api/...8h_source.html.

    I don't know if the license suits you though. But even then you can implement such proxy model yourself, I've done that once or twice and it worked.
    Your biological and technological distinctiveness will be added to our own. Resistance is futile.

    Please ask Qt related questions on the forum and not using private messages or visitor messages.


Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.