PyQt5 SegFault when adding rows to QTreeView using QSortFilterProxyModel

[atlas]

the project is called Vivisect and it has been using PyQt5. Vivisect is a binary analysis framework (eg. a disassembler)
this is a relatively stable project, but while recently attempting to add in filtering capabilities to various QTreeView widgets using QSortFilterProxyModel, i'm getting somewhat inconsistent segfaults, with very little context (ie. none other than printed debug statements).

i've looked through this thread but i haven't been able to stop the segfaulting completely: https://www.qtcentre.org/threads/692...Source-segfalt

when i start up analysis, it always crashes (it populates the various widgets):

$ vivbin /bin/chown                                                                                                                                                                                               
Loaded (0.0685 sec) /bin/chown                                                                                                                                                                                                                                            
Beginning analysis...                                                                                                                                                                                                                                                     
...analyzing exports.                                                                                                                                                                                                                                                     
0x0200ad31: Emulation Found 0x0200afd0 (from func: 0x0200ad20) via call 0x0200afd0                                                                                                                                                                                        
libpng warning: iCCP: known incorrect sRGB profile                                                                                                                                                                                                                        
libpng warning: iCCP: known incorrect sRGB profile                                                                                                                                                                                                                        
...                                                                                                                                                                                                           
Dynamic Branch found at 0x2001d48    call rax                                                                                                                                                                                                                             
Segmentation fault (core dumped)

$ vivbin /bin/chown                                                                                                                                                                                               
Loaded (0.0685 sec) /bin/chown                                                                                                                                                                                                                                            
Beginning analysis...                                                                                                                                                                                                                                                     
...analyzing exports.                                                                                                                                                                                                                                                     
0x0200ad31: Emulation Found 0x0200afd0 (from func: 0x0200ad20) via call 0x0200afd0                                                                                                                                                                                        
libpng warning: iCCP: known incorrect sRGB profile                                                                                                                                                                                                                        
libpng warning: iCCP: known incorrect sRGB profile                                                                                                                                                                                                                        
...                                                                                                                                                                                                           
Dynamic Branch found at 0x2001d48    call rax                                                                                                                                                                                                                             
Segmentation fault (core dumped)

To copy to clipboard, switch view to plain text mode 

here's the relative code snippits (links to full code listed below):

class VivFilterModel(QSortFilterProxyModel):
     def __init__(self, parent=None):
         QSortFilterProxyModel.__init__(self, parent=parent)
         self.setDynamicSortFilter(True)
         self.setFilterKeyColumn(-1)
 
     def __getattr__(self, name):    # the existing "navModel" includes a couple functions
         return getattr(self.sourceModel(), name)
 
 
class VivFilterView(QWidget):
     '''
     This is the primary window for the VQViv*Views if they want to include filtering
     '''
     window_title = '__undefined__'
     view_type = None
 
     def __init__(self, vw, vwqgui, *args, **kwargs):
         QWidget.__init__(self)
 
         self.view = self.view_type(vw, vwqgui, *args, **kwargs)
         self.ffilt = VQFilterWidget(self)
 
         layout = vq_basics.VBox(self.view, self.ffilt)
         self.setLayout(layout)
 
         self.ffilt.filterChanged.connect(self.textFilterChanged)
         self.setWindowTitle(self.view.window_title)
 
     def textFilterChanged(self):
         regExp = QtCore.QRegExp(self.ffilt.text(), 
                                 self.ffilt.caseSensitivity(),
                                 self.ffilt.patternSyntax())
 
         self.view.filterModel.setFilterRegExp(regExp)
 
class VQTreeModel(QtCore.QAbstractItemModel):
    '''
    A QT tree model that uses the tree API from visgraph
    to hold the data...
    '''
 
    columns = ( 'A first column!', 'The Second Column!')
    editable = None
    dragable = False
 
    def __init__(self, parent=None, columns=None):
 
        if columns != None:
            self.columns = columns
 
        QtCore.QAbstractItemModel.__init__(self, parent=parent)
        self.rootnode = VQTreeItem((), None)
 
        if self.editable == None:
            self.editable = [False,] * len(self.columns)
 
    def vqEdited(self, pnode, col, value):
        return value
 
    def append(self, rowdata, parent=None):
        if parent == None:
            parent = self.rootnode
 
        pidx = self.createIndex(parent.row(), 0, parent)
        i = len(parent.children)
        self.beginInsertRows(pidx, i, i)
        node = parent.append(rowdata)
        self.endInsertRows()
        self.layoutChanged.emit()
        return node
...
    def sort(self, colnum, order=0):
        cmpf = VQTreeSorter(colnum, order)
        self.layoutAboutToBeChanged.emit()
        self.rootnode.children.sort(cmp=cmpf)
        self.layoutChanged.emit()
 
    def flags(self, index):
        if not index.isValid():
            return 0
        flags = QtCore.QAbstractItemModel.flags(self, index)
        col = index.column()
        if self.editable[col]:
            flags |= QtCore.Qt.ItemIsEditable
        if self.dragable:
            flags |= QtCore.Qt.ItemIsDragEnabled
        return flags
 
    def columnCount(self, parent=None):
        return len(self.columns)
 
    def data(self, index, role):
        if not index.isValid():
            return None
 
        item = index.internalPointer()
        if role == QtCore.Qt.DisplayRole:
            return item.data(index.column())
 
        if role == QtCore.Qt.UserRole:
            return item
 
        return None
 
    def setData(self, index, value, role=QtCore.Qt.EditRole):
 
        node = index.internalPointer()
        if not node:
            return False
 
        # If this is the edit role, fire the vqEdited thing
        if role == QtCore.Qt.EditRole:
            value = self.vqEdited(node, index.column(), value)
            if value == None:
                return False
 
        node.rowdata[index.column()] = value
        self.dataChanged.emit(index, index)
 
        return True
 
    def headerData(self, column, orientation, role):
        if ( orientation == QtCore.Qt.Horizontal and
             role == QtCore.Qt.DisplayRole):
 
            return self.columns[column]
 
        return None
 
    def index(self, row, column, parent):
 
        if not self.hasIndex(row, column, parent):
            return QtCore.QModelIndex()
 
        pitem = parent.internalPointer()
        if not pitem:
            pitem = self.rootnode
 
        item = pitem.child(row)
        if not item:
            return QtCore.QModelIndex()
 
        return self.createIndex(row, column, item)
 
    def parent(self, index):
 
        if not index.isValid():
            return QtCore.QModelIndex()
 
        item = index.internalPointer()
        if not item:
            return QtCore.QModelIndex()
 
        pitem = item.parent
 
        if pitem == self.rootnode:
            return QtCore.QModelIndex()
 
        if pitem == None:
            return QtCore.QModelIndex()
 
        return self.createIndex(pitem.row(), 0, pitem)
...

class VivFilterModel(QSortFilterProxyModel):
     def __init__(self, parent=None):
         QSortFilterProxyModel.__init__(self, parent=parent)
         self.setDynamicSortFilter(True)
         self.setFilterKeyColumn(-1)

     def __getattr__(self, name):    # the existing "navModel" includes a couple functions
         return getattr(self.sourceModel(), name)


class VivFilterView(QWidget):
     '''
     This is the primary window for the VQViv*Views if they want to include filtering
     '''
     window_title = '__undefined__'
     view_type = None

     def __init__(self, vw, vwqgui, *args, **kwargs):
         QWidget.__init__(self)

         self.view = self.view_type(vw, vwqgui, *args, **kwargs)
         self.ffilt = VQFilterWidget(self)

         layout = vq_basics.VBox(self.view, self.ffilt)
         self.setLayout(layout)

         self.ffilt.filterChanged.connect(self.textFilterChanged)
         self.setWindowTitle(self.view.window_title)

     def textFilterChanged(self):
         regExp = QtCore.QRegExp(self.ffilt.text(), 
                                 self.ffilt.caseSensitivity(),
                                 self.ffilt.patternSyntax())

         self.view.filterModel.setFilterRegExp(regExp)

class VQTreeModel(QtCore.QAbstractItemModel):
    '''
    A QT tree model that uses the tree API from visgraph
    to hold the data...
    '''

    columns = ( 'A first column!', 'The Second Column!')
    editable = None
    dragable = False

    def __init__(self, parent=None, columns=None):

        if columns != None:
            self.columns = columns

        QtCore.QAbstractItemModel.__init__(self, parent=parent)
        self.rootnode = VQTreeItem((), None)

        if self.editable == None:
            self.editable = [False,] * len(self.columns)

    def vqEdited(self, pnode, col, value):
        return value

    def append(self, rowdata, parent=None):
        if parent == None:
            parent = self.rootnode

        pidx = self.createIndex(parent.row(), 0, parent)
        i = len(parent.children)
        self.beginInsertRows(pidx, i, i)
        node = parent.append(rowdata)
        self.endInsertRows()
        self.layoutChanged.emit()
        return node
...
    def sort(self, colnum, order=0):
        cmpf = VQTreeSorter(colnum, order)
        self.layoutAboutToBeChanged.emit()
        self.rootnode.children.sort(cmp=cmpf)
        self.layoutChanged.emit()

    def flags(self, index):
        if not index.isValid():
            return 0
        flags = QtCore.QAbstractItemModel.flags(self, index)
        col = index.column()
        if self.editable[col]:
            flags |= QtCore.Qt.ItemIsEditable
        if self.dragable:
            flags |= QtCore.Qt.ItemIsDragEnabled
        return flags

    def columnCount(self, parent=None):
        return len(self.columns)

    def data(self, index, role):
        if not index.isValid():
            return None

        item = index.internalPointer()
        if role == QtCore.Qt.DisplayRole:
            return item.data(index.column())

        if role == QtCore.Qt.UserRole:
            return item

        return None

    def setData(self, index, value, role=QtCore.Qt.EditRole):

        node = index.internalPointer()
        if not node:
            return False

        # If this is the edit role, fire the vqEdited thing
        if role == QtCore.Qt.EditRole:
            value = self.vqEdited(node, index.column(), value)
            if value == None:
                return False

        node.rowdata[index.column()] = value
        self.dataChanged.emit(index, index)

        return True

    def headerData(self, column, orientation, role):
        if ( orientation == QtCore.Qt.Horizontal and
             role == QtCore.Qt.DisplayRole):

            return self.columns[column]

        return None

    def index(self, row, column, parent):

        if not self.hasIndex(row, column, parent):
            return QtCore.QModelIndex()

        pitem = parent.internalPointer()
        if not pitem:
            pitem = self.rootnode

        item = pitem.child(row)
        if not item:
            return QtCore.QModelIndex()

        return self.createIndex(row, column, item)

    def parent(self, index):

        if not index.isValid():
            return QtCore.QModelIndex()

        item = index.internalPointer()
        if not item:
            return QtCore.QModelIndex()

        pitem = item.parent

        if pitem == self.rootnode:
            return QtCore.QModelIndex()

        if pitem == None:
            return QtCore.QModelIndex()

        return self.createIndex(pitem.row(), 0, pitem)
...

To copy to clipboard, switch view to plain text mode 

the working pull request is here: https://github.com/vivisect/vivisect/pull/237/files
the full branch is here: https://github.com/atlas0fd00m/vivis...filtered_views

i expect the bug is some simple thing i'm doing (eg. not providing a parent, or maybe connecting signals in incompatible ways, etc...)
but i'm at a complete loss. i don't understand why adding in a QSortFilterProxyModel would cause the entire python interpreter to segfault.

thank you in advance for any help and guidance you can provide? i'm relatively new to pyqt and qt in general, so don't worry about offending me.
let me know if you need any other context (or clarification)

@

---

Added after 9 minutes:

i ran out of room for these bits of relevant code.

class VQVivFunctionsViewPart(VQVivTreeView):
 
    _viv_navcol = 0
    window_title = 'Functions'
    columns = ('Name','Address', 'Size', 'Ref Count')
 
 
     def __init__(self, vw, vwqgui):
        VQVivTreeView.__init__(self, vw, vwqgui, withfilter=True)
        self.navModel = VivNavModel(self._viv_navcol, self, columns=self.columns)
        self.filterModel = VivFilterModel()
        self.filterModel.setSourceModel(self.navModel)
        self.setModel(self.filterModel)
 
        self.vqLoad()
        self.vqSizeColumns()
 
class VQVivFunctionsView(VivFilterView):
    view_type = VQVivFunctionsViewPart
 
class VQTreeView(QTreeView):
 
    def __init__(self, parent=None, cols=None, **kwargs):
        QTreeView.__init__(self, parent=parent, **kwargs)
        self.setSortingEnabled(True)
        self.setAlternatingRowColors(True)
 
        if cols != None:
            model = VQTreeModel(parent=self, columns=cols)
            self.setModel( model )
 
    def vqSizeColumns(self):
        c = self.model().columnCount()
        for i in xrange(c):
            self.resizeColumnToContents(i)
 
    def setModel(self, model):
        model.dataChanged.connect( self.dataChanged )
        model.rowsInserted.connect( self.rowsInserted )
        return QTreeView.setModel(self, model)
 
class EnviNavModel(vq_tree.VQTreeModel):
 
    dragable = True
 
    def __init__(self, navcol, parent=None, columns=None):
        vq_tree.VQTreeModel.__init__(self, parent=parent, columns=columns)
        self.navcol = navcol
 
    def mimeData(self, idx):
        pnode = idx[0].internalPointer()
        expr = pnode.rowdata[self.navcol]
        mdata = QtCore.QMimeData()
        mdata.setData('envi/expression',str(expr))
        return mdata

class VQVivFunctionsViewPart(VQVivTreeView):

    _viv_navcol = 0
    window_title = 'Functions'
    columns = ('Name','Address', 'Size', 'Ref Count')


     def __init__(self, vw, vwqgui):
        VQVivTreeView.__init__(self, vw, vwqgui, withfilter=True)
        self.navModel = VivNavModel(self._viv_navcol, self, columns=self.columns)
        self.filterModel = VivFilterModel()
        self.filterModel.setSourceModel(self.navModel)
        self.setModel(self.filterModel)

        self.vqLoad()
        self.vqSizeColumns()

class VQVivFunctionsView(VivFilterView):
    view_type = VQVivFunctionsViewPart

class VQTreeView(QTreeView):

    def __init__(self, parent=None, cols=None, **kwargs):
        QTreeView.__init__(self, parent=parent, **kwargs)
        self.setSortingEnabled(True)
        self.setAlternatingRowColors(True)

        if cols != None:
            model = VQTreeModel(parent=self, columns=cols)
            self.setModel( model )

    def vqSizeColumns(self):
        c = self.model().columnCount()
        for i in xrange(c):
            self.resizeColumnToContents(i)

    def setModel(self, model):
        model.dataChanged.connect( self.dataChanged )
        model.rowsInserted.connect( self.rowsInserted )
        return QTreeView.setModel(self, model)

class EnviNavModel(vq_tree.VQTreeModel):

    dragable = True

    def __init__(self, navcol, parent=None, columns=None):
        vq_tree.VQTreeModel.__init__(self, parent=parent, columns=columns)
        self.navcol = navcol

    def mimeData(self, idx):
        pnode = idx[0].internalPointer()
        expr = pnode.rowdata[self.navcol]
        mdata = QtCore.QMimeData()
        mdata.setData('envi/expression',str(expr))
        return mdata

To copy to clipboard, switch view to plain text mode 

if you are looking through the codebase, the PyQt5 files of interest should be:

vivisect/qt/views.py
vqt/tree.py
envi/qt/memory.py
vqt/common.py
vivisect/qt/main.py

i realize it's a little complex, but i appreciate the help, and i'm happy to provide whatever context may make it easier to troubleshoot.

thanks!
@

[atlas]

i don't know if it's at all related, but since migrating to PyQt5 (from PyQt4), i've been getting a ton of these messages:

QXcbConnection: XCB error: 8 (BadMatch), sequence: 3577, resource id: 190841046, major code: 130 (Unknown), minor code: 3

QXcbConnection: XCB error: 8 (BadMatch), sequence: 3577, resource id: 190841046, major code: 130 (Unknown), minor code: 3

To copy to clipboard, switch view to plain text mode 

(running Linux/Kubuntu 18.04)

[anda_skoa]

The filter model should not cause any problem by itself, but it might be triggering bugs in the source model.

The code looks pretty good though.
I can see only one unchecked access of an internal pointer: in data() when returning the value for DisplayRole.

You also don't need to emit layoutChanged() in append, begin/endInsertRows() handle that correctly.

I see you are connecting to the model's dataChanged() and rowsInserted() signals in your view.
Maybe it is holding some data that gets invalidate when the filter removes entries?

Cheers,
_

[atlas]

thank you for your insights! i somehow missed your post. i'll have to check my notifications.

i'll look into that. i also had another user hit me off-list, recommending a change to a couple lines which call model() but which should also directly access sourceModel():

in "vivisect/qt/views.py":

- idx = self.model().createIndex(pnode.row(), col, pnode)
+ idx = self.model().sourceModel().createIndex(pnode.row() , col, pnode)
# We are *not* the edit role...
- self.model().setData(idx, val, role=QtCore.Qt.DisplayRole)
+ self.model().sourceModel().setData(idx, val, role=QtCore.Qt.DisplayRole)
(thanks Norm!)

hopefully these pointers can help identify the instability.
again, i appreciate your insights!
i'll let you know what i find.

---

Added after 17 minutes:

You also don't need to emit layoutChanged() in append, begin/endInsertRows() handle that correctly.

I see you are connecting to the model's dataChanged() and rowsInserted() signals in your view.
Maybe it is holding some data that gets invalidate when the filter removes entries?

_

if i remove the call to emit, nothing added to the view. i end up with an empty view. i'm not doubting what you're saying about begin/endInsertRows(), i'm wondering how my code is doing odd stuff which is keeping it from operating correctly. and if that may also cause my problems.
as for holding data that gets invalidated when the filter removes entries, unfortunately the segfaults are occurring before i get to filter anything.

thanks again!

[atlas]

The filter model should not cause any problem by itself, but it might be triggering bugs in the source model.

The code looks pretty good though.
I can see only one unchecked access of an internal pointer: in data() when returning the value for DisplayRole.

You also don't need to emit layoutChanged() in append, begin/endInsertRows() handle that correctly.

I see you are connecting to the model's dataChanged() and rowsInserted() signals in your view.
Maybe it is holding some data that gets invalidate when the filter removes entries?

Cheers,
_

i'm having pretty good luck with the changes requested and no longer segfaulting....
i'm now struggling with dramatic delays since adding the filter, and i think it may have to do with my event handling.
i'm trying to remove the self.layoutChanged.emit() and everything is much faster, but the QTreeView displays nothing. i can tell that there's data in the model, because filtering takes a while (this test bin has 20000 entries in the tree) to filter. i've tried connecting things differently, but i'm not seeing changes. i've connected the model to some debugging functions, and i can *see* that rowsAboutToBeInserted and rowsInserted signals are emitted, but nothing seems to cause the QTreeView to visually show any of the data.

how can i get the QTreeView to adjust what data it thinks is there without emitting layoutChanged?

i tried going back to the code pre-filtering and removing the emits, and have similar troubles. however, oddly, i get a few entries listed. i can see that there are way more items than listed, because when i change from sort-ascending to sort-descending, the items are completely different ones (ie. i'm looking at but a window of the available entries). is there some way to get QTreeView to refresh how many items it's supposed to be displaying?

help? thanks in advance

[anda_skoa]

This is very strange, layoutChanged() should definitely not be needed when adding or removing rows.

When you say "going back to pre-filtering", do you mean you set the main model directly on the view?
And it still doesn't behave correctly?

Cheers,
_

[atlas]

This is very strange, layoutChanged() should definitely not be needed when adding or removing rows.

When you say "going back to pre-filtering", do you mean you set the main model directly on the view?
And it still doesn't behave correctly?

Cheers,
_

correct, that's what i meant.

however, long story short, i removed it from a different branch (without filtering) but left in two seemingly minor changes, and things *seem* to work properly.

https://github.com/atlas0fd00m/vivis...e/atlas_merged
(vqt/tree.py and vivisect/qt/views.py)

the filter_views branch is here:
https://github.com/atlas0fd00m/vivis...filtered_views


i just had to back out the filter_views branch because, while i no longer segaulted, the gui never come up when filtering. i could get the gui to come up if i removed the layoutChanged.emit() line, but then none of the QTrees showed their data.

any ideas? it seems that we've worked the code around so that the layoutChanged.emit() causes problems with and without filters it some warped way, that feels like progress.

@

btw, i apologize again for the delay in response. i just discovered that i wasn't subscribed to this thread corrected.