You will have to use some media player backend: On Windows, DirectShow, on Mac Quicktime, on Linux either Mplayer or xine. There are may be others, but these are the most common I think.

You would then get a window handle for a QWidget and pass it to these engines to play a video in that widget.

Edit: What is your GTK-thing? Maybe you can embed that into Qt. But why do you want a QDialog when there is already a GTK solution?