Try

LIBS += -L/path/to/library -lLibrary_name_without_the_first_lib

I am assuming that the library is libmathlib.so.

Carlos.