Writing numbers to file fast! (Need Faster float to char implementation)

[drhex]

The quickest way to do it is not to convert floats to text

Do you have any control over the application that is supposed to read the data you write, i.e. could you write the raw bits that make up the double values without converting them to text and have the reader accept them in that format?

[philwinder]

Do you have any control over the application that is supposed to read the data you write

Yes I have full control, since I am the only user. At the moment I usually import the data into scilab or matlab to do some math, so I can essentially accept anything that isnt encoded in some wierd encrypted format.

What were you thinking? Using one of the toHex() routines and write that? Since I dont think sprintf has a binary option?

[drhex]

I was thinking that you could cast a pointer to the first element of your double array to char * and feed it along with the size of the array in bytes to QIODevice::write

[philwinder]

drhex, pgorszkowski:
I have now tried your suggestions (you were suggesting the same thing) and they work fantastically. Writing the data to a binary format is many times faster than before.

The allocation of the array takes about 140ms, which I dont think will get any faster now (although I might try and use QVector to see how that compares - done takes 310 ms). However the more interesting bit is that a QFile::write takes 1 second, whereas the std::ofstream takes 3 seconds. I thought the std function would have been faster, but there you go. Maybe we really are to a point where hard drive speed is the limiting factor.

By the way, allocating an array then writing the whole array is faster than writing each individual element (1.2 seconds vs 3 seconds) which I guess one would instinctively assume.

Many thanks to drhex and pgorszkowski and as always, wysota. Cheers.

P.s.
For testing and future viewers, here is some cleaned up code with the fastest possible combination:

QTime timer; 
 
QMessageBox::information(0,"","Ready");
QFile file;
file.setFileName("C:/test.txt");
file.open(QFile::WriteOnly);
 
double f = 0.541263266;
 
int size = 200000*30;
double * af = new double[size];
 
timer.start();
for( int x = 0; x < size; x++ )
	af[x] = f;
 
int time = timer.elapsed();
QMessageBox::information(0,"","Elapsed Time: " + QString::number(time) + "ms" );
timer.restart();
file.write((const char*)af, size*sizeof(double));
file.flush();
file.close();
time = timer.elapsed();
QMessageBox::information(0,"","Elapsed Time: " + QString::number(time) + "ms" );
delete [] af;

QTime timer; 
 
QMessageBox::information(0,"","Ready");
QFile file;
file.setFileName("C:/test.txt");
file.open(QFile::WriteOnly);
	
double f = 0.541263266;

int size = 200000*30;
double * af = new double[size];

timer.start();
for( int x = 0; x < size; x++ )
	af[x] = f;

int time = timer.elapsed();
QMessageBox::information(0,"","Elapsed Time: " + QString::number(time) + "ms" );
timer.restart();
file.write((const char*)af, size*sizeof(double));
file.flush();
file.close();
time = timer.elapsed();
QMessageBox::information(0,"","Elapsed Time: " + QString::number(time) + "ms" );
delete [] af;

To copy to clipboard, switch view to plain text mode 

[d_stranz]

With these performance statistics on 30 seconds worth of data (1.2 s), it looks like you might be able to write your data in real time, as opposed to storing 30 seconds worth and then writing. Also would give you some protection against data loss if something fails - can you afford to lose 30 seconds worth of data? Writing in real time, you lose at most a second or so.

Not that you should think in units of seconds in the first place - if you are sampling a continuous data stream at 200kS/s, then -the smallest- time interval at which you can write in real time without data loss would be better than buffering for some human-defined period like 1, 5, or 30 seconds before writing.

This is where you might make use of a memory-mapped file. If you know how long you will be acquiring data in total (say 10 minutes), then you allocate the memory-mapped file to a size of 200000 * 600 * sizeof ( double ) and simply acquire to that "array" as if it were the double * array you allocate in your example code. The file system takes care of flushing it to disk at appropriate times (or you can do it manually every second or whatever if you don't trust the file system to do it for you).

With memory-mapping, you can completely ignore all the details of file I/O except those involved in originally opening and then closing the memory-mapped file. Everything else just looks like filling up an array.

David

[drhex]

I notice that you tried earlier to multiply your aquired values with 1000000 to get an integer. How much precision does your data have? Using float rather than integer might be sufficient, letting you store twice as many values or write them faster.

[philwinder]

Hi guys,
I have been using the previous routine to save data now for quite a few months, and everything seems to be fine in Real time.

@d_stranz: Ok, but the reason I had to move to a file anyway was because my system could only handle a minutes worth of data in memory. Since there are 4 channels, sampling at 200kS/s = 800kS/s * 32 bit for float ~= 24.4mB/s. One minute = 1.5 Gig. A lot of data.

Also, I cannot find a qt related class that deals with it, so would I have to write my own?

@drhex: Thats was just because I thought that saving an int would be smaller/quicker. I now know that was wrong. The final piece of code is what I have been using. (Or something similar)