If I allow Qt to handle moving the selection handles, then the handles and parent move as one object.
I dont think so . You must be setting the movable flag on the parent.
I tried a simple example where I added child graphicsitem in a parent graphics item. I set the setFlags(QGraphicsItem::ItemIsMovable) on the child item, and it works fine for me. Only the child moves.

I also set movable flag on the parent, so when i click parent, all the child items move with it, but when i move the item whose flag was set to movable, it moves. I also had other items which were fixed in the parent item.

Please try a small example urself and u will understand