Hello,

my aim is to read very large files (10GB to 30GB) into main memory (my MacPro has 32GB installed, running OSX 10.5.8).

This is basically very easy to achieve using Qt:

Qt Code: 
  1. QFile file(filename);
  2. file.open(QIODevice::ReadOnly);
  3. file.readAll();
  4. file.close();
To copy to clipboard, switch view to plain text mode 

However, what happens on OSX is that the so called Unified Buffer Cache (UBC) automatically caches all reads and when reading a file larger than 16GB the systems ends up with 16GB of the file in the active memory and 16GB of the same data in the inactive memory (the cache) and starts to swap heavily. This is a well known problem and the UBC can be disabled with a simple additional system call:

Qt Code: 
  1. #include <fcntl.h>
  2. QFile file(filename);
  3. file.open(QIODevice::ReadOnly);
  4. fcntl(file.handle(), F_NOCACHE, 1); // <- disable UBC for that file
  5. file.readAll();
  6. file.close();
To copy to clipboard, switch view to plain text mode 

This successfully disables caching and allows loading files up to 30GB into main memory.

However, the problem now is that reading performance with the cache disabled drastically drops from ~200mb/s to ~25mb/s.

Does anyone have any hints on how to have the UBC disabled but still achieve high reading throughput?

Best regards & thanks a lot,
mikeee7