is there a reason that you chose to add the label etc into the QWidget and not into to graphicsscene?
The labels I add (using QLabel) are the plot title and other things that are outside of the canvas region. They have no relationship to the data being displayed - they are just decorations added by the user to give the plot a title or a footnote. ("My Beautiful and Informative Plot")

Labels for individual points in the canvas area are added as part of the scene. These have the same scaling issues as the dots - you generally don't want the point labels to grow as the plot is zoomed in. In addition, it is very awkward to dimension the point labels in world coordinates (when the data range is 0 - 1000 degrees C by 0 - 100 kPa, exactly what world coordinate size should you pick for a 10 pt label for a dot at 100 degrees C and 10 kPa anyway?), so I fake this by creating the dot labels with pixel coordinates and positioning them in world coordinates. When the plot is zoomed in, I scale the text so that it remains the same size. Eg. when the plot is zoomed in (a smaller data range per pixel) then text is zoomed inversely so that its pixel size remains constant.