Many problems can be solved with enough levels of indirection. Consider adding another layer in between your semaphore app and your worker:

Attach the semaphore to another qt thread. Let that thread (can be blocked by semaphore) communicate with the worker (non blocking)

also let the gui communicate with the worker (signal/slot) to indicate cancelling.