Why doesn't Qt have an async local file API and what to do instead?

[numbat]

What do people do when they need to read a lot of data from a local file without blocking the main thread? Is the only method to fire up another thread? There is QNetworkAccessManager which will work with local files but it only does whole files. Any ideas? Many people must have faced this problem, or are hard disks fast enough that it usually doesn't matter?

Has anybody used QNetworkAccessManager for this purpose? Any issues I should be aware of, if I use it?

[tbscope]

Reading and writing to a file is synchronous when you read or write the data in one continuous block. If you do that, as you know, it will block everything else till it is done. For small files, this delay is neglectful but for large files you might want to read in blocks.

QFile lets you do that. Create a file reader or writer that writes in "chunks" rather than the whole thing at once.
You can even use QTextStream or QDataStream to do that.

Edit: Note though that QFile doesn't implement the readyRead() signal for example.
I would create a file reading class that implements a signal that says that a chunk has been read or can be written.
Then I would connect this signal to a slot and keep reading or writing chunks.
In the mean time, your program has time to process other events.

Alternatively, as you said, you could use threads too, but be careful, you need to add a great deal of extra code to prevent multiple threads accessing the same file at the same time for example (among other things)

[squidge]

I've used threads to do file i/o before as the file i/o I was doing was typically over a network which could be slow (ie, it could take several seconds depending on network load) and I didn't want to application to show the "Not responding" message because I wasn't processing events.

Generally however, I don't bother. File i/o is infrequent enough (and mostly local based) to not worry about.

[numbat]

Tbscope and Fatjuicymole, thanks a lot. Tbscope, I like your idea of chunks, but I'm not sure when to read them, because as you not QFile does not emit readyRead. I suppose I could use a zero timer, so that the event loop is pumped between each chunk read. However, this means the disk will not be operating while events are being processed. For maximum performance, I'm leaning towards using the chunk idea, but in a separate dedicated thread. Thanks again.

[numbat]

OK, I did a rough implementation. It doesn't yet contain error handling or cleanup, but I was interested in what you think. The basic idea is a single thread that caches file handles and serves read requests as they come in (via the queued signal/slot).
Usage:

FileReader fr;
    fr.start();
 
    FileReaderFile frf(&fr, "/home/somefile");
    connect(&frf, SIGNAL(dataAvailable(QByteArray, int, int)), SLOT(dataAvailableEvt(QByteArray, int, int)));
    frf.addReadRequest(0, 100000);
    frf.addReadRequest(100000, 100000);

FileReader fr;
    fr.start();

    FileReaderFile frf(&fr, "/home/somefile");
    connect(&frf, SIGNAL(dataAvailable(QByteArray, int, int)), SLOT(dataAvailableEvt(QByteArray, int, int)));
    frf.addReadRequest(0, 100000);
    frf.addReadRequest(100000, 100000);

To copy to clipboard, switch view to plain text mode 

Implementation:

/*
 * FileReaderFile
 */
FileReaderFile::FileReaderFile(FileReader *fr, QString filename) :
        m_filename(filename), m_fr(fr)
{
    connect(this, SIGNAL(addFile(QString,FileReaderRequest*)), fr, SLOT(addFileEvt(QString, FileReaderRequest*)));
}
 
 
void FileReaderFile::addReadRequest(int start, int count)
{
     FileReaderRequest * fri = new FileReaderRequest(m_fr, start, count);
     connect(fri, SIGNAL(dataAvailable(QByteArray, int, int)), this, SIGNAL(dataAvailable(QByteArray, int, int)));
     emit addFile(m_filename, fri);
}
 
/*
 * FileReaderRequest
 */
FileReaderRequest::FileReaderRequest(FileReader *fr, int start, int count) :
        m_start(start), m_count(count)
{
    moveToThread(fr);
}
 
bool FileReaderRequest::process(QFile *file)
{
    /* Read the request, then delete ourselves. */
    file->seek(m_start);
    QByteArray ba = file->read(m_count);
    emit dataAvailable(ba, m_start, ba.length());
    deleteLater();
    return true;
}
 
/*
 * FileReader
 */
FileReader::FileReader()
{
    moveToThread(this);
}
 
void FileReader::addFileEvt(const QString & filename, FileReaderRequest *fri)
{
    QFile * file;
    if (m_files.contains(filename))
    {
        file = m_files.value(filename);
    }
    else
    {
        file = new QFile(filename);
        file->open(QIODevice::ReadOnly);
        m_files.insert(filename, file);
    }
 
    fri->process(file);
}
 
void FileReader::run()
{
    exec();
}

/*
 * FileReaderFile
 */
FileReaderFile::FileReaderFile(FileReader *fr, QString filename) :
        m_filename(filename), m_fr(fr)
{
    connect(this, SIGNAL(addFile(QString,FileReaderRequest*)), fr, SLOT(addFileEvt(QString, FileReaderRequest*)));
}


void FileReaderFile::addReadRequest(int start, int count)
{
     FileReaderRequest * fri = new FileReaderRequest(m_fr, start, count);
     connect(fri, SIGNAL(dataAvailable(QByteArray, int, int)), this, SIGNAL(dataAvailable(QByteArray, int, int)));
     emit addFile(m_filename, fri);
}

/*
 * FileReaderRequest
 */
FileReaderRequest::FileReaderRequest(FileReader *fr, int start, int count) :
        m_start(start), m_count(count)
{
    moveToThread(fr);
}

bool FileReaderRequest::process(QFile *file)
{
    /* Read the request, then delete ourselves. */
    file->seek(m_start);
    QByteArray ba = file->read(m_count);
    emit dataAvailable(ba, m_start, ba.length());
    deleteLater();
    return true;
}

/*
 * FileReader
 */
FileReader::FileReader()
{
    moveToThread(this);
}

void FileReader::addFileEvt(const QString & filename, FileReaderRequest *fri)
{
    QFile * file;
    if (m_files.contains(filename))
    {
        file = m_files.value(filename);
    }
    else
    {
        file = new QFile(filename);
        file->open(QIODevice::ReadOnly);
        m_files.insert(filename, file);
    }

    fri->process(file);
}

void FileReader::run()
{
    exec();
}

To copy to clipboard, switch view to plain text mode