Originally Posted by
jesse_mark
do you use
readyRead ,readyReadStandardError, readyReadStandardOutput signals ??
this will be trigger every time the buffer is full or its flushed.
I have no control over the compressor's source code, so I don't know how he retrieves from stdin. I'm guessing he waits for some sort of flush (which QFile has but QProcess does not), but I could be wrong.
Added after 1 14 minutes:
I've found a solution to my problem.
Rather than writing directly to the QProcess, I instead write to a QByteArray via a QBuffer and then manually flush the QBuffer into the QProcess at regular intervals. I was able to reduce my file's write speed from the awful 170 seconds I was seeing to approximately 4 seconds, only about 50% slower than running it as a stand-alone process.
It basically boils down to the fact that the buffer built into QProcess is designed for transmitting relatively short strings between processes, not large data blocks. The result was that the buffer got flushed WAY more often than it needed to.
In the end, my write speeds are about the same as I would have gotten writing to the disk first and running it in stand-alone mode after the fact, but at least I avoid the intermediate files.
Sample code:
/* set up compressor sub-process */
connect(compressor, SIGNAL(readyReadStandardError()), this, SLOT(retrieveStdErr()));
connect(compressor, SIGNAL(readyReadStandardOutput()), this, SLOT(retrieveStdOut()));
qDebug("Starting compressor...");
compressor
->start
("d:/tools/compressor.exe", args,
QIODevice::ReadWrite);
if(!compressor->waitForStarted(3000))
{
qWarning("Could not start compressor. Aborting write.");
delete compressor;
return false;
}
/* QProcess doesn't have a big enough buffer - set an intermediate one */
zip_buffer.setBuffer(&dataBuffer);
/* start processing */
int interval = 50000;
for(int i = 0; i < ThingList.size(); ++i)
{
const Thing& it = ThingList.at(i);
io << it;
/* flush the buffer after writing a lot of things */
if(i % interval == 0)
{
compressor->write(zip_buffer.data());
zip_buffer.seek(0);
}
}
compressor->write(zip_buffer.data());
/* give the compressor time to catch up */
if(!compressor->waitForFinished(5000))
{
qWarning("Too slow!);
return false;
}
compressor->deleteLater();
return true;
/* set up compressor sub-process */
QProcess* compressor = new QProcess();
connect(compressor, SIGNAL(readyReadStandardError()), this, SLOT(retrieveStdErr()));
connect(compressor, SIGNAL(readyReadStandardOutput()), this, SLOT(retrieveStdOut()));
qDebug("Starting compressor...");
compressor->start("d:/tools/compressor.exe", args, QIODevice::ReadWrite);
if(!compressor->waitForStarted(3000))
{
qWarning("Could not start compressor. Aborting write.");
delete compressor;
return false;
}
/* QProcess doesn't have a big enough buffer - set an intermediate one */
QByteArray dataBuffer;
QBuffer zip_buffer;
zip_buffer.setBuffer(&dataBuffer);
zip_buffer.open(QIODevice::ReadWrite);
QDataStream io(&zip_buffer);
/* start processing */
int interval = 50000;
for(int i = 0; i < ThingList.size(); ++i)
{
const Thing& it = ThingList.at(i);
io << it;
/* flush the buffer after writing a lot of things */
if(i % interval == 0)
{
compressor->write(zip_buffer.data());
zip_buffer.seek(0);
}
}
compressor->write(zip_buffer.data());
/* give the compressor time to catch up */
if(!compressor->waitForFinished(5000))
{
qWarning("Too slow!);
return false;
}
compressor->deleteLater();
return true;
To copy to clipboard, switch view to plain text mode
Bookmarks