You are reading two bytes every milisecond without checking if those two bytes are really there to read.
Qt Code:
  1. if(port->bytesAvailable())
To copy to clipboard, switch view to plain text mode 
I guess you have here -1 most of the time, but still you're processing the data.
I think 1ms is to often, you're just wasting your OS resources.