Need concurrency advice with callback function

[BreakBad]

(PyQt 4.8, Python 2.7)
After reading through the following I'm still left a bit puzzled as to which approach to attempt to utilize for my situation. Any advice would be appreciated.
Keeping_the_GUI_Responsive

I have a module, flp.so library to which I have no control over. It takes a function as a constructor argument, and performs a complex/long calculation when invoking the run() method. Here I subclassed QThread, and invoke flp.run inside the thread's run method. In turn the flp module will invoke the callback method, which emits a signal. I catch the signal back in my main thread, and update a model for a QTreeView and update a QProgressBar.

def my_runner(QThread):
    def __init__(self,callback=None):
        self.callback = callback
 
    def callback(self):
        # f_runner.f_output is array that is appended every iteration of f_runner.run()
        emit(SIGNAL('update_main'),self.f_runner.f_output.pop())
 
    def run(self):
        import flp
        self.f_runner = flp.f_runner(callback=self.callback)
        self.f_runner.run()

def my_runner(QThread):
    def __init__(self,callback=None):
        self.callback = callback

    def callback(self):
        # f_runner.f_output is array that is appended every iteration of f_runner.run()
        emit(SIGNAL('update_main'),self.f_runner.f_output.pop())

    def run(self):
        import flp
        self.f_runner = flp.f_runner(callback=self.callback)
        self.f_runner.run()

To copy to clipboard, switch view to plain text mode 

However, the GUI is not very responsive, as the signals are probably stacking up and the events can't be processed fast enough. I try to avoid invoking QApplication.processEvents(), but tried it here and very bad things happen.

I have been rethinking this design but have gotten stuck. Eventually I would like the main thread GUI to be responsive while this thread is working and updating the progress bar and table view in the main.

TYVM for reading.

[wysota]

I'd have to have some more info to give you any precise hints. Currently I can only advise to batch updates to the model so that the view doesn't have to redraw itself each time you add a row to the model.

[BreakBad]

Thanks for the response!

The only thing thats happening on the main thread side is connecting the update signal to this slot, where I've just been tinkering with repaint, etc....:

self.connect(my_runner,SIGNAL('update_main'),self.update_main)
....
def update_main(self,data):
        self.run_progress_bar.increment()
        #self.run_progress_bar.repaint()
        row = self.output_model.rowCount()
        for col in range(len(data)):
            item = QtGui.QStandardItem(data[col])
            #self.output_model.beginInsertRows(QtCore.QModelIndex(),row,row)
            self.output_model.setItem(row,col,item)
            #self.output_model.endInsertRows()

self.connect(my_runner,SIGNAL('update_main'),self.update_main)
....
def update_main(self,data):
        self.run_progress_bar.increment()
        #self.run_progress_bar.repaint()
        row = self.output_model.rowCount()
        for col in range(len(data)):
            item = QtGui.QStandardItem(data[col])
            #self.output_model.beginInsertRows(QtCore.QModelIndex(),row,row)
            self.output_model.setItem(row,col,item)
            #self.output_model.endInsertRows()

To copy to clipboard, switch view to plain text mode 

The runs could be anywhere from a few rows to 1,000 rows, each row taking anywhere from a fraction of a second to half a second to calculate. Eventually my 'customers' would like to also have multiple runs going concurrently. We have 16 processor/core workstations and would like to utilize them. So perhaps using Runnables would be better?

I agree, that batching the rows of data would be much better. I'll look into creating a consumer thread with a timer *edit: instead of a producer with a signal

[wysota]

If your GUI gets frozen then provided that worker threads are running correctly, the problem is with refreshing views thus batching is something that can give you the biggest speedup. I suggest you try it and see for yourself.