writing huge binary data

[Alex22]

Hi,
there is a 2D array 500*100,000 like this:

double (*data)[100000];
data = new double[500][100000];
 
for(int i=0; i<500; i++)
{
    for(int j=0; j<100000; j++)
    {
        if(j==0)
        {
            data[i][j]=0xdd00bb00;//i want this to be my header
            continue;
        }
        data[i][j] = (i+1)*(j+1);
    }       
}
// now data was initialized

double (*data)[100000];
data = new double[500][100000];

for(int i=0; i<500; i++)
{
    for(int j=0; j<100000; j++)
    {
        if(j==0)
        {
            data[i][j]=0xdd00bb00;//i want this to be my header
            continue;
        }
        data[i][j] = (i+1)*(j+1);
    }       
}
// now data was initialized

To copy to clipboard, switch view to plain text mode 

I wrote this code:

void write(QString filename)
{
    QFile myfile(filename);
 
    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug()<<"can not open to write";
        return;
    }
 
    QDataStream out(&myfile);
 
                                     //now I want to write all of "data" in binary mode
 
    myfile.flush();
    myfile.close();
}

void write(QString filename)
{
    QFile myfile(filename);

    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug()<<"can not open to write";
        return;
    }

    QDataStream out(&myfile);
    
                                     //now I want to write all of "data" in binary mode

    myfile.flush();
    myfile.close();
}

To copy to clipboard, switch view to plain text mode 

thanks for any help

[anda_skoa]

Well, if you don't have any format constraints, just loop over the entries and write them.
Maybe write the dimensions before, so you can read them before reading the data, unless your data has defined fixed size.

Cheers,
_

[Alex22]

please read under post (sorry because of slow speed of the sit post was been repeated)!!!

[Alex22]

@anda_skoa thanks. when i use this to write, it gives a filename.txt in zero size:

double (*data)[100000];
    data = new double[500][100000];
 
    for(int i=0; i<500; i++)
    {
        for(int j=0; j<100000; j++)
        {
            if(j==0)
            {
                data[i][j]=0xdd00bb00;//i want this to be my header
                continue;
            }
            data[i][j] = (i+1)*(j+1);
        }       
    }
    // now data was initialized
 
    QFile myfile("d:/filename.txt");
 
    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug()<<"can not open to write";
        return;
    }
 
    QDataStream out(&myfile);
 
   out.writeRawData((char *) (&(data[0][0])), 100000*500*sizeof(double)); // did you mean i write here??? but .txt size is zero and nothing is written
 
    myfile.flush();
    myfile.close();

double (*data)[100000];
    data = new double[500][100000];
     
    for(int i=0; i<500; i++)
    {
        for(int j=0; j<100000; j++)
        {
            if(j==0)
            {
                data[i][j]=0xdd00bb00;//i want this to be my header
                continue;
            }
            data[i][j] = (i+1)*(j+1);
        }       
    }
    // now data was initialized

    QFile myfile("d:/filename.txt");
 
    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug()<<"can not open to write";
        return;
    }
 
    QDataStream out(&myfile);
 
   out.writeRawData((char *) (&(data[0][0])), 100000*500*sizeof(double)); // did you mean i write here??? but .txt size is zero and nothing is written

    myfile.flush();
    myfile.close();

To copy to clipboard, switch view to plain text mode 

thanks for any help

[anda_skoa]

Well, since you indicated usage of QDataStream, I assumed you wanted to use it, so I meant looping through data and writing the doubles.

Now the question is, why instantiate a QDataStream if you are then bypassing it?
If you want to go down the route of doing a memory dump, why not just use QFile directly?

In any case, have you checked the return values of the operations? Also what size does QFile report after writing and after closing?

Cheers,
_

P.S.: naming a binary file ".txt" doesn't make sense.

[Alex22]

@anda_skoa, thanks. I found an amazing thing happens here. for data[m][n] (double (*data)[n]; data = double[m][n]) when m>n it gives me right myfile.txt but when m<n it gives a file in size of zero!!! for example when m=599 and n=600 it gives me a file in size of zero and returns:"The program has unexpectedly finished." but when m=601 and n=600 it gives me a file in size of 2,344kB

even when m is so larger than n, there is no false but if n is only one bigger that m, its not ok!!!

for example:

#include <QApplication>
#include <QFile>
#include <QDebug>
#include <QDataStream>
 
 
int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
 
    QFile myfile("d:/test.txt");
 
    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug() << "can not open to write";
        return 1;
    }
 
    double (*data)[600];
    data = new double[599][600];//now is not ok but if i define data = new double[601][600] it gives me right file!!!
 
        for(int i=0; i<599; i++)
        {
            for(int j=0; i<600; i++)
            {
                data[i][j] = (i+1)*(j+1);
            }
            myfile.write((char *)(&(data[i][600])), sizeof(double));
        }
 
   //myfile.write((char *)(&(data[0][0])), 599*600*sizeof(double)); when m>n this is true also
 
    myfile.flush();
    myfile.close();
 
    return 0;
}

#include <QApplication>
#include <QFile>
#include <QDebug>
#include <QDataStream>


int main(int argc, char *argv[])
{
    QApplication a(argc, argv);

    QFile myfile("d:/test.txt");

    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug() << "can not open to write";
        return 1;
    }

    double (*data)[600];
    data = new double[599][600];//now is not ok but if i define data = new double[601][600] it gives me right file!!!

        for(int i=0; i<599; i++)
        {
            for(int j=0; i<600; i++)
            {
                data[i][j] = (i+1)*(j+1);
            }
            myfile.write((char *)(&(data[i][600])), sizeof(double));
        }

   //myfile.write((char *)(&(data[0][0])), 599*600*sizeof(double)); when m>n this is true also

    myfile.flush();
    myfile.close();

    return 0;
}

To copy to clipboard, switch view to plain text mode 

You are right I need no QDataStrem and I can use QFile directly.

thanks for anymore help

[anda_skoa]

and returns:"The program has unexpectedly finished."

Ah, no.
It crashed.

You could run the program in the debugger and look where it crashes.

I would suggest to either use a one dimensional array and calculate the offset based on i and j or initialize each entry in your array of rows individually.

Cheers,
_

[Alex22]

Thanks Anda,
I wrote only in one row and now it gives a file in size of 390,625kB (for below example), but almost every byte inside of this file is zero. my code is:

#include <QApplication>
#include <QFile>
#include <QDebug>
 
int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
 
    QFile myfile("d:/test.txt");
 
    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug() << "can not open to write";
        return 1;
    }
 
    double *d;
    d = new double[500*100000];
 
    int i = 0;
    while(i<100000)
    {
        for(int j=0; j<500; j++)
        {
            d[j]=(i+1)*(j+1);
        }
    i = i + 500;
    }
 
    myfile.write((char *) d, 500*100000*sizeof(double));
    myfile.flush();
    myfile.close();
 
    return 0;
}

#include <QApplication>
#include <QFile>
#include <QDebug>

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);

    QFile myfile("d:/test.txt");

    if(!myfile.open(QIODevice::WriteOnly))
    {
        qDebug() << "can not open to write";
        return 1;
    }

    double *d;
    d = new double[500*100000];

    int i = 0;
    while(i<100000)
    {
        for(int j=0; j<500; j++)
        {
            d[j]=(i+1)*(j+1);
        }
    i = i + 500;
    }

    myfile.write((char *) d, 500*100000*sizeof(double));
    myfile.flush();
    myfile.close();

    return 0;
}

To copy to clipboard, switch view to plain text mode 

Thanks for your helping

[anda_skoa]

You are always writing to the first 500 doubles.

Cheers,
_

[Alex22]

@anda_sko, sorry I must write [j+(i*500)] and i++. now all thing is OK and i could write from a 2D array to a text.