QVector crashes when array size is big

[Sheng]

#include<QApplication>
#include<iostream>
#include<QString>
#include<QVector>
 
struct myStruc
{
 QString name;
 int array[3000*2000];
};
 
int main(int argc,char* argv[])
{
  QApplication app(argc,argv);
  QVector<myStruc> myVector;
  myStruc currentStruc;
  for (int i=0;i<5;i++)
   {
     currentStruc.name+="a";
     currentStruc.array[0]=i;
     myVector.push_back(currentStruc);
   }
  for(int i=0;i<5;i++)
   {
     std::cout<<myVector[i].name.toStdString()<<"   "<<myVector[i].array[0]<<std::endl;
   }
 myVector.clear();
 return app.exec();
}

#include<QApplication>
#include<iostream>
#include<QString>
#include<QVector>

struct myStruc
{
 QString name;
 int array[3000*2000];
};

int main(int argc,char* argv[])
{
  QApplication app(argc,argv);
  QVector<myStruc> myVector;
  myStruc currentStruc;
  for (int i=0;i<5;i++)
   {
     currentStruc.name+="a";
     currentStruc.array[0]=i;
     myVector.push_back(currentStruc);
   }
  for(int i=0;i<5;i++)
   {
     std::cout<<myVector[i].name.toStdString()<<"   "<<myVector[i].array[0]<<std::endl;
   }
 myVector.clear();
 return app.exec();
}

To copy to clipboard, switch view to plain text mode 

Compile fine, got "segmentation fault" error when run it.
It works OK if I decrease array size to 3000, any solutions?
Thanks in advance.

[abernat]

Have you stepped through it in a debugger or added qDebug() output to determine where you're getting the seg fault?

[talk2amulya]

what happens when u try other container classes like QList?

[Sheng]

what happens when u try other container classes like QList?

Same thing happens (QVector,QList,QLinkedList, std::vector, std::list). Did some search and found this looks like a notorious problem of STL container. I guess Qt Container might use STL during implementation since STL Containers crash as well.
Anyway, I will write a simple list container tomorrow.

[Sheng]

Have you stepped through it in a debugger or added qDebug() output to determine where you're getting the seg fault?

The segmentation fault comes when push_back function is called.

[talk2amulya]

well, the issue is that QVector and other containers allocate CONTIGUOUS memory..and ur every array takes 24+MB of memory plus QString which is also dynamic..so there are definitely chances that when u try to push back one more..it being another contiguous location enters into another segment..thus a segmentation fault..a solution to this would be PROBABLY create a container that doesn't allocate contiguous memory but spreaded..but still its risky cuz amount being allocated is huge...maybe some guru can give a better insight on this

[wysota]

6*3000*2000*sizeof(int) = 36M*sizeof(int) = 144 (or 288) MB just for creating the array. Aren't you running out of memory? If not, are you sure C++ copies arrays? I doubt that so copying the struct might lose its integrity.

[lni]

The crash is not because it is QVector. You program will crash without QVector in pure C++..

Don't put such a huge array in stack. You should use memory in the heap. But it is going to swap if you don't have enough memory...

Do this:

struct myStruc
{
 QString name;
 int *array;
 
  myStruc() : array( new int[ 3000*2000] ) {
  }
 
  // or with size argument in constructor
  myStruct( size_t sz ) : array( new int[ sz ] ) {
  }
 
  ~myStruc() {
     delete [] array;
  }
 
};

struct myStruc
{
 QString name;
 int *array;

  myStruc() : array( new int[ 3000*2000] ) {
  }

  // or with size argument in constructor
  myStruct( size_t sz ) : array( new int[ sz ] ) {
  }

  ~myStruc() {
     delete [] array;
  }

};

To copy to clipboard, switch view to plain text mode 

[wysota]

Actually QVector allocates its data on the heap even if the vector object itself is on the stack.

[Sheng]

6*3000*2000*sizeof(int) = 36M*sizeof(int) = 144 (or 288) MB just for creating the array. Aren't you running out of memory? If not, are you sure C++ copies arrays? I doubt that so copying the struct might lose its integrity.

the Container will crash even I only push_back one object.

[drhex]

QVector probably allocates memory for several objects at a time, to avoid having to reallocate at every push_back(). How much RAM is there in your testbox, Sheng?

Does it work if you supply an initial vector size, e.g:

QVector<myStruc> myVector(5);

QVector<myStruc> myVector(5);

To copy to clipboard, switch view to plain text mode 

[Sheng]

QVector probably allocates memory for several objects at a time, to avoid having to reallocate at every push_back(). How much RAM is there in your testbox, Sheng?

at least 1G.
It does not work either with reserve or resize.

[Sheng]

Just wrote and tested my own linked list, it works fine.

[wysota]

Just wrote and tested my own linked list, it works fine.

Which proves exactly nothing.

[Sheng]

Which proves exactly nothing.

It at least demonstrated the problem is not from the limitation of the system, but from the container.

[drhex]

I've experimented some more and abbreviated the program:

#include<QVector>
 
struct myStruc {
    int array[3000*2000];
};
 
int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);
 
    QVector<myStruc> myVector;
    myStruc currentStruc;
    myVector.push_back(currentStruc);
}

#include<QVector>

struct myStruc {
    int array[3000*2000];
};

int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);

    QVector<myStruc> myVector;
    myStruc currentStruc;
    myVector.push_back(currentStruc);
}

To copy to clipboard, switch view to plain text mode 

For array-sizes upto 3000*232 (=2.8MB), the above program works as expected.

For array-sizes between 3000*233 and 3000*698, it prints "hello" and then segfaults.

For array-sizes 3000*699 (=8.4MB) and up, it segfaults immediately, without even printing "hello".

Adding a QApplication object did not help.

Tested with gcc 4.3.2 on Ubuntu 8.10 using Qt-4.5.0-beta1.

Sheng, you've found the struct-of-death!

[Sheng]

I've experimented some more and abbreviated the program:

#include<QVector>
 
struct myStruc {
    int array[3000*2000];
};
 
int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);
 
    QVector<myStruc> myVector;
    myStruc currentStruc;
    myVector.push_back(currentStruc);
}

#include<QVector>

struct myStruc {
    int array[3000*2000];
};

int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);

    QVector<myStruc> myVector;
    myStruc currentStruc;
    myVector.push_back(currentStruc);
}

To copy to clipboard, switch view to plain text mode 

For array-sizes upto 3000*232 (=2.8MB), the above program works as expected.

For array-sizes between 3000*233 and 3000*698, it prints "hello" and then segfaults.

For array-sizes 3000*699 (=8.4MB) and up, it segfaults immediately, without even printing "hello".

Adding a QApplication object did not help.

Tested with gcc 4.3.2 on Ubuntu 8.10 using Qt-4.5.0-beta1.

Sheng, you've found the struct-of-death!

Nice research, thanks.

[drhex]

#include <stdio.h>
#include <unistd.h>
 
struct myStruc {
    int array[3000*2000];
};
 
 
void foo(const myStruc &v)
{
    printf("%d\n", v.array[1]);
}
 
int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);
 
    myStruc currentStruc;
    foo(currentStruc);
}

#include <stdio.h>
#include <unistd.h>

struct myStruc {
    int array[3000*2000];
};


void foo(const myStruc &v)
{
    printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);

    myStruc currentStruc;
    foo(currentStruc);
}

To copy to clipboard, switch view to plain text mode 

The above also crashes without printing hello, and now there isn't any Qt code left! Seems like a compiler or OS problem.

[Sheng]

#include <stdio.h>
#include <unistd.h>
 
struct myStruc {
    int array[3000*2000];
};
 
 
void foo(const myStruc &v)
{
    printf("%d\n", v.array[1]);
}
 
int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);
 
    myStruc currentStruc;
    foo(currentStruc);
}

#include <stdio.h>
#include <unistd.h>

struct myStruc {
    int array[3000*2000];
};


void foo(const myStruc &v)
{
    printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);

    myStruc currentStruc;
    foo(currentStruc);
}

To copy to clipboard, switch view to plain text mode 

The above also crashes without printing hello, and now there isn't any Qt code left! Seems like a compiler or OS problem.

try use heap to see what happens. Should work fine, then try to push back to the container.
Here is what I got:
hello
0
segmentation fault

[drhex]

#include <stdio.h>
#include <sys/resource.h>
 
int main(int , char *[])
{
    struct rlimit rlim;
    getrlimit(RLIMIT_STACK, &rlim);
    printf("max stack size=%d bytes\n", (int)rlim.rlim_cur);
}

#include <stdio.h>
#include <sys/resource.h>

int main(int , char *[])
{
    struct rlimit rlim;
    getrlimit(RLIMIT_STACK, &rlim);
    printf("max stack size=%d bytes\n", (int)rlim.rlim_cur);
}

To copy to clipboard, switch view to plain text mode 

This prints

max stack size=8388608 bytes

on my machine. One learns something every day!