Hi I figured out what's been bugging me.

See I used the the setSceneRect(0,0,100,100) to set the scene. What I didn't figure out what the it created the 100x100 scene in the middle of the QGraphicsViewport and everything (position and size was relative to that). Which now makes sense.

So how did I solved it? I just did setSceneRect(0,0,this->width(),this->height()); Then every position and size got refered to the scene which coincided with the viewport (in this case I'm referring to the QGraphicsView widget as the viewport).

Thanks for all the help.