detecting that a file is currently being written to...

[momesana]

I need to write a programm that monitors a directory, and upon the arrival of a file, opens it, parses the text, does something with it and moves it to another directory afterwards. During testing the code I encountered that sometimes the content read was empty even though there should be some data.

After a few hours I found out, that the QFileSystemWatcher emits the signal directoryChanged() as soon as a file start being written to the directory. In case I try to open the file at that very moment, the returned QByteArray is empty. If the file has completely been written to disk however, there is no problem .

How can I work around this? I could wait a few seconds before opening the file but this is an ugly workaround especially since the file may be larger and a few seconds may not suffice in that case ...


Thanx in advance

Edit: What I need is a way to make sure no-one else is writing to the file, before I attempt to open it.

[high_flyer]

Well, you can try to open it for writing and reading - this will fail as long it is being written by someone else.

[kernel_panic]

you could wait until the file is writable with that:

QFileInfo::isWritable ()

QFileInfo::isWritable ()

To copy to clipboard, switch view to plain text mode 

but beware, if its write protected it will end in an infinite loop.
or you do that for waiting some seconds:

public slots:
		void fileProcessing(QString&);
 
void MyApp::fileProcessing(QString & filePath)
{
	//Do sth with the file
}
 
void MyApp::fileModified(QString & filePath)
{
	QTimer::singleShot(5000, this, SLOT(fileProcessing(filePath)));
}

public slots:
		void fileProcessing(QString&);
		
void MyApp::fileProcessing(QString & filePath)
{
	//Do sth with the file
}

void MyApp::fileModified(QString & filePath)
{
	QTimer::singleShot(5000, this, SLOT(fileProcessing(filePath)));
}

To copy to clipboard, switch view to plain text mode 

[momesana]

I tried a combination of:

/* Open file. return false if open fails. */
	if (!input.open(QIODevice::Text | QIODevice::ReadWrite)) {
		qDebug() << "couldn't open file ReadWrite" << input.fileName();
		return false;
	}
 
	/* Check the size of the file. If its zero, return */
	if (!input.isWritable()) {
		qDebug() << "Skipping file" << endl;
		return false;
	}

/* Open file. return false if open fails. */
	if (!input.open(QIODevice::Text | QIODevice::ReadWrite)) {
		qDebug() << "couldn't open file ReadWrite" << input.fileName();
		return false;
	}
	
	/* Check the size of the file. If its zero, return */
	if (!input.isWritable()) {
		qDebug() << "Skipping file" << endl;
		return false;
	}

To copy to clipboard, switch view to plain text mode 

but it doesn't work. That means I am opening the file before the content is written by the copy command and reading out an empty QByteArray half of time (depending on the speed of the Operating system. Sometimes the file is already written when I open the QFile and in that case the file can be read properly). :'(

I am using Linux btw, though the code is supposed to be run on both environments (windoze,linux). Right now, the only alternative I see is checking the QFileInfo lastModified field to check whether the file is older then 5 minutes or so and then handle the file. Ugly workaround especially since operating on remote files requires syncing the clock of the two machines or keeping a QFileInfo map and refresh and compare the values to see whethere they were changed since the last time checked or not.

[^NyAw^]

Hi,

Is the program that insert the files your?

[momesana]

Hi,

Is the program that insert the files your?

The files are written somewhere and copied to the directory I am monitoring. It is a simple copy as far as I know.

The process is as follows:
Medical Data is being analysed by a program that then writes the results of the analysis as a PCL File to the mentioned directory. I open the files and if necessary split them into individual files according to a simple pattern. Then I write these resulting files into a directory of its own where it is fetched by another application (a fax application). Then this data is being downloaded by medical offices via modems.

Consequently it is very very important that nothing gets screwed up or lost during the process.

[kernel_panic]

i dont know if it works when other applications are writing data into the file, but try

QFile::waitForBytesWritten ( int msecs )

QFile::waitForBytesWritten ( int msecs )

To copy to clipboard, switch view to plain text mode 

[momesana]

i dont know if it works when other applications are writing data into the file, but try

QFile::waitForBytesWritten ( int msecs )

QFile::waitForBytesWritten ( int msecs )

To copy to clipboard, switch view to plain text mode 

What if file is being just written, a few bytes are there, but the file is not complete yet? This way I would end up with a truncated file.

[^NyAw^]

Hi,

So you don't had wirtten the program that is copying the files, you only wirte a program that is monitoring them. I'm right?

So you could try to make a bucle that only gets out when the file size is the same at two consecutive iterations, waiting a little time evrey iteration.

[momesana]

Ok, simple and ugly workaround:
I forget about QFileSystemWatcher and use QTimer. Every five minutes I read the entries in the source directory and save them into a variable but before I do that, I process the entries read by the last timerEvent. The first time the application is launched the list is empty so all I do is saving the entries. Five minutes later I process these entries and save a new entrylist generated by QDir::entryList(). This way every file processed is at least 5 minutes old (unless the file was overwritten after being queued in the list but I think that will not happen) and the I assume the copy process will never take more than five minutes. If it really happens, though, that is an act of fate and the patient should die! :-P

[kernel_panic]

uuuuh!Yeah! Very ugly workaround.... it must be possible to do this without timers...
I looked into the docu and just saw qbuffer. if you use the readyRead signal and do than your file processing, it could workbut im not sure...
but this version with the timers.... i wouldnt use it for medical apps.... it has to be secure.

[momesana]

Yep. Indeed very ugly. I will read about QBuffer when I get to work in half an hour. Thanks