So there's a main thread a a worker thread. Only the main thread can change config settings; the worker thread objects only read from it.

Thread safety is not an issue here. The main thread makes its changes, then tells the worker thread objects a change has occurred. Maybe you want some kind of lock so the main thread won't change any values while the worker objects update themselves, but that isn't clear. If you roll your own notification system, similar to Java's event registration, you don't even need that; the main thread simply waits until the call to update the workers returns before it does anything else. There's probably a similar mechanism available through Qt's signal/slot framework. But a vector of pointers to every worker object is all that's needed here.

Store references in it if you can't get over your irrational dislike of pointers. You'll still be using pointers, but the syntax will let you fool yourself into thinking you're not.