Threading for a video output window?

[DivideByZer0]

Hi, I'm a complete beginner at threading in any programming language, and I was trying to set up threading with a video stream that I'm displaying in an application I'm building for a research project using PyQt4. The frames are acquired through OpenCV, then converted to a pixmap and displayed in a QGraphicsScene as part of a QGraphicsView. The QGraphicsView also uses the drawForeground function to draw an overlay (a fairly simple grid and some text) on the video stream. The video is a little bit choppy but that's alright for now.

The issue I'm having is that the application needs to interface with an external device connected through the serial port every so often. During this process, the data that is used to update the foreground changes as does the contents of the video stream. What I want to do is have the video stream and the drawForeground function (but primarily the video stream) running continually as this external device performs its procedure (which can take 30 seconds to a couple minutes, as of now.) Right now I've gotten it to update the stream mid-procedure in a kind of hacky way, passing the app object reference down to the class that performs the serial functions and having it call app.processEvents() every time it gets a break in communication. However, this is jumpy and we would like to get a continual stream running independently of the serial communication.

I've read up a little bit about threading and while it seems like it would be able to accomplish what we're trying to do here, I'm having some trouble figuring out where to start and I don't fully understand the concepts. I will be glad to provide specific information about my app, and any tutorials or advice would be greatly appreciated!

[stampede]

I think the best option for you is to wrap the serial port code into QObject subclass, run it in separate thread (with "moveToThread") and communicate with it by signals and slots.
Another way could be to use QtConcurrent module, read the documentation for details (examples are included).

The frames are acquired through OpenCV, then converted to a pixmap and displayed in a QGraphicsScene

Btw. if you are capturing with OpenCV, consider using QImage instead of QPixmap, I think it can be more efficient (you can skip the conversion - QPainter has methods to draw QImages).

[DivideByZer0]

Thanks, I will definitely look into this. The Serial code extends the Serial class, and then it has a class between it and the main app that performs more high-level operations - do you think it would be appropriate to make this interface class a QObject and run it in a separate thread?

I'll also look into QtConcurrent.

Also, I was wondering how I'd implement a QImage as a video in a QGraphicsView/Scene - I poked around a bit trying to figure this out but I haven't quite found how to add it as the content of the scene. If you know anything about this off the top of your head it would be a great help, I've been trying to figure out how to optimize the stream for awhile - we were previously using VideoCapture with PyGame and it was running a lot more smoothly.

[wysota]

Is there any particular reason why you are using QGraphicsView? If you are just showing the video and possibly some overlay text or other simple shapes then it might be faster (in terms of performance) if you used a custom widget instead of graphics view which adds to the complexity without giving you any practical advantages over other solutions. Another thing to look into is to render the video as a texture using OpenGL.

Btw. if you are capturing with OpenCV, consider using QImage instead of QPixmap, I think it can be more efficient (you can skip the conversion - QPainter has methods to draw QImages).

Correct me if I'm wrong but doesn't QPainter convert QImage to QPixmap before drawing to screen?

[stampede]

do you think it would be appropriate to make this interface class a QObject and run it in a separate thread?

I don't know. I'd just think about moving any code that executes in blocking manner into QObject subclass and run it in separate thread.

Also, I was wondering how I'd implement a QImage as a video in a QGraphicsView/Scene

Use the QPainter::drawImage method to display the data captured with OpenCV (google for "IplImage to QImage" if you don't know how to convert).

Correct me if I'm wrong but doesn't QPainter convert QImage to QPixmap before drawing to screen?

I don't know.
If yes, you should try the Qt OpenGL support.

[DivideByZer0]

Oh, I'm using QGraphicsView because it has a little more straightforward functionality than the QGL widget that I'd looked into using, and DrawForeground was very convenient for the overlay portion - no major reasons. Our main goal was to quickly get a prototype app up and running with the basic functionality we wanted, and we're pretty much there at this point, so conversion to QGL might be an option. Right now we're getting a pretty steady 9FPS when the image is still, but it drops to <1FPS when the image moves (it's a 1344x1024 stream from a lab camera).

One issue with that is we are using one brand new computer with an excellent graphics card, but the other PC we are working on is an older Pentium 4 without much of a graphics card (the QImage>QPixmap method ran more efficiently on that PC than a basic QGL implementation we had put together to test performance). We could probably update the graphics card in that PC fairly easily though.

[wysota]

Then I'd suggest to write a simple QWidget subclass that will display the image and the overlay instead of using graphics view. For just two-three objects using graphics view is an overkill.

[DivideByZer0]

Hey guys, thanks for the help! I ended up just using threading.Thread with a separate class for each command, and hacked together a simple queue out of a list, so that commands don't get executed concurrently. The video feed is actually running very smoothly right now, it's not perhaps as perfect as it could be, but it's not choppy anymore which is all we need. Since the classes we wrote to interface with the serial device weren't written as QObjects, and there aren't really simultaneous data access issues, this has worked fine. Thanks again for your help and advice, it will be very useful as we continue improving our application!