Did you tried with non blocking communication? I think slowdowns can be caused by synchronization of both peers.

You may try to post events from first peer to 2 thread with some events buffer, that would deal with communication with 2 thread of 2 peer(also with buffer) so overloaded connection would slowdown only additional threads (while there would still be a number of events in buffers). In such approach only additional threads would be blocking and peers would post/receive events to/from buffers in non blocking way...

p.s.
dialogs uses local loops and main event loop is theoretically suspended(maybe it's processed rarely from dialog's local loop)... are you sure that second peer receives also modal dialog's events? Because if it is, there are some differences between qt manual and real implementation.