I am using fontconfig now and just pass the family name, style and size to it. Works now!
Thanks
Markus