1. Font scaling with resolution
Keep in mind that on Windows and maybe on other OS, users can change font magnification at the OS level. On Windows, Qt abides by this, so a 10 pt font at 125% magnification is equivalent to a 12.5 pt font at 100%. I have seen this cause clipping and other layout issues if other widgets (like QLabel) have size constraints set which limit their expansion. If no constraints are set, then with magnification the widgets will expand to hold the full text.

2. Custom layout for QFileDialog
Is it possible to just change the position of certain elements in the QFileDialog?
I'm in Linux and I want the "Open" button on the bottom right and the file extension on top.
On some OS, the OS native file dialog is used instead of a Qt-specific one, so it may not be possible to customize. You may be able to override this using QFileDialog::setOptions() before the dialog is shown on screen or derive a custom dialog from QFileDialog. To get to the individual widgets inside the Qt-based QFileDialog, you will need to call QFileDialog::layout() (inherited from QWidget) and then try to figure out which widget corresponds to the things you want to modify.

Sounds like more trouble than it is worth and a likely source of confusion to users who are used to seeing a particular style on their OS.