Ok to get the fact straight, the model() function is not const, I must have been tired when I first look at the documentation. So I tried setting the supportedDracAction on it and it worked when startDrag was called but for some reason, when dragEnter, dragMove and dropEvent were called, I always got the proposed action to be Qt::CopyAction which seemed weird when I set Qt::MoveAction.

QDrag::start functions add Qt::CopyAction no matter what suggested drop actions were. I then continued to trace and got to the platform specific implementation which I didn't want to understand. I chose Qt for a reason afterall

So after trying some stuff, I needed to reimplement dropEvent only to set the right dropAction based on drag source because in dropMimeData, there is no way that I know of to know what is the drag source.

So that was my solution.