Can't read large file (2.5GB) with QFile

[enricong]

I am trying to read a 2.5GB text file.

QFile input_file(filename);
    if (!input_file.open(QIODevice::ReadOnly))
        return;
    qint64 x = input_file.bytesAvailable();
    QString test = input_file.readLine();
    QString test2 = input_file.readAll();

QFile input_file(filename);
    if (!input_file.open(QIODevice::ReadOnly))
        return;
    qint64 x = input_file.bytesAvailable();
    QString test = input_file.readLine();
    QString test2 = input_file.readAll();

To copy to clipboard, switch view to plain text mode 

Both test and test2 are empty.
x = 2682500677
Also, isReadble() is true, but canReadLine() is false
I can read a 5MB file with no problem.

[ChrisW67]

QIODevice::readLine() and QIODevice::readAll() return QByteArrays not QStrings. The QString is being constructed using the conversion constructor, which uses QString::fromAscii(). Depending on the current codec if the first byte of the file is not valid, e.g. a UTF-8 byte-order-mark, then you may well end up with an empty string.

So, what does you file actually contain, and is it valid characters only?


...
On second thoughts, even if your byte array slurped the entire file into memory, conversion to 16-bit characters in a QString will double the size and blow the 4GB limit on 32-bit architectures.

[DanH]

Yeah, I'm guessing he's out of storage, one way or another.

[enricong]

I am on a 64-bit RHEL5 system

The file is all text.

Upon further examination, readLine() does work. the first line in the file is blank, if I read the second line, I could see the text. Although this seems to get stuck once the program hits 1.0gb of resident memory usage.

readAll() just seems to give up and not even try. I looked into it more closely and it looks like it fails to resize the QByteArray to 2.5gb. I would have expected an error but I guess that makes sense.
I've got 12GB ram on this system.

[SixDegrees]

What is the output of 'ulimit -a'? And what context does your code occur in - main() or within a function?

[DanH]

The amount of RAM you have on the system has little to do with this. A 4-byte number can represent a bit over 4g, and many environments limit things to half that so that arithmetic will be signed. In Qt the array size and object length fields are 4 bytes long.

It's very poor programming practice to read an entire file larger than a megabyte or so anyway. It causes storage thrashing and really mucks up performance.

---

Added after 9 minutes:

Of course, the other option would be to convert the file to a SQL database.

[zimang]

Is your mem enough? Maybe the way is on another.