is there anyway to avoid this?
As a developer you need much of the database software API in order to build the Qt plugin. Thereafter you should need only the client library.

As a deployed client you typically need only the client library. Some vendors provide a convenient client-only installer but I don't think that's the case for PostgreSQL. For PostgreSQL (Windows) you should just be able to deploy libpq.dll with your application.