I think that your problem is that you are using viewport coordinates in your mouse events instead of scene coordinates. You are even using mapFromScene() but passing it viewport coordinates instead of scene coordinates.

BTW. I suggest you take a look at QGraphicsView::dragMode.