QVector crashes when array size is big

[lni]

myStruc* currentStruc=new myStruct is in heap, but the element inside myStruct is in stack!

[Sheng]

myStruc* currentStruc=new myStruct is in heap, but the element inside myStruct is in stack!

I do not think so, in my own linked list, I use this and works fine. The maximum stack size is around 8M. the imageData is larger than 10M.

I defined my structure as follows:

struct imageStruc
{
 QString fileName;
 uint8_t imageData[IMAGE_SIZE];
 imageStruc* next;
};

struct imageStruc
{
 QString fileName;
 uint8_t imageData[IMAGE_SIZE];
 imageStruc* next;
};

To copy to clipboard, switch view to plain text mode 

[wysota]

but the element inside myStruct is in stack!

No, it's not. If it were on the stack and you returned from the function the frame of which it would be positioned in, the data would get destructed which is of course not the case.

What is on the stack is probably the local copy of the structure when push_back is called (although I'm not sure as push_back probably takes a const reference and not a copy).

Sheng: Why do you need that array there in the first place? Can't you use a vector that dynamically allocates its data as it is needed?

By the way, if you don't need a vector but a linked list, use QLinkedList and not QVector

[Sheng]

No, it's not. If it were on the stack and you returned from the function the frame of which it would be positioned in, the data would get destructed which is of course not the case.

What is on the stack is probably the local copy of the structure when push_back is called (although I'm not sure as push_back probably takes a const reference and not a copy).

Sheng: Why do you need that array there in the first place? Can't you use a vector that dynamically allocates its data as it is needed?

By the way, if you don't need a vector but a linked list, use QLinkedList and not QVector

Yes, actually I used linked list, just use QVector for the example since all of them crashes.

[wysota]

Yes, actually I used linked list, just use QVector for the example since all of them crashes.

Still you didn't say what is the array meant for.

In his example, the stack is contained inside the heap.

I'm sorry, but are you sure you know what the stack is? Because based on your post I think you don't.

[lni]

No, it's not. If it were on the stack and you returned from the function the frame of which it would be positioned in, the data would get destructed which is of course not the case.
QLinkedList and not QVector

In his example, the stack is contained inside the heap. The stack will only get destructed when the heap is deleted, it will not go out of scope if the heap is no deleted. When you do "new myStruct" to construct the object, where does the array[ 2000x300 ] come from? It comes from the stack, and it will crash immediately at the "new myStruct".

As I said, the element inside the structure is in the stack, change it to heap and it should work.

[Sheng]

In his example, the stack is contained inside the heap. The stack will only get destructed when the heap is deleted, it will not go out of scope if the heap is no deleted. When you do "new myStruct" to construct the object, where does the array[ 2000x300 ] come from? It comes from the stack, and it will crash immediately at the "new myStruct".

As I said, the element inside the structure is in the stack, change it to heap and it should work.

no, it's not, and it works in my code. the crash comes not from "new myStruct", but from push_back.

[lni]

no, it's not, and it works in my code. the crash comes not from "new myStruct", but from push_back.

Your computer may have enough resource to hold 1 copy of array[ 2000 x 3000 ], and crash if you have two copies, that is precisely what QVector does, it copies your structure when you do push_back the object (value based object).

Go back review my post on calling the destructor and copy constructor prinout, and how you add a copy constructor to fix the crash.

[Sheng]

Your computer may have enough resource to hold 1 copy of array[ 2000 x 3000 ], and crash if you have two copies, that is precisely what QVector does, it copies your structure when you do push_back the object (value based object).

Go back review my post on calling the destructor and copy constructor prinout, and how you add a copy constructor to fix the crash.

There are more than 10 images in the memory, not just two.

[fullmetalcoder]

myStruc* currentStruc=new myStruct is in heap, but the element inside myStruct is in stack!

AFAIK if you allocate a struct on heap ALL its field can only be allocated on heap...

Now some thoughts that may or may not be relevant :

• big structures like that should never ever be passed by value anywhere... allocate them on heap and use a QVector<myStruct*> instead of a QVector<myStruct> (or whatever container you may like more)
• Storing the data directly in the struct instead of a pointer to it is a HUGE design mistake IMO because it forces it to be allocated no matter what and that gets even worse when some copy has to be made. Using a pointer and allocating the data when it is needed would probably be a better idea, would allow sharing and would very probably cause a dramatic decrease of memory usage and a dramatic increase in speed
• huge contiguous arrays of memory are not a very wise design choice. Maybe you can split that data in smaller pieces? And do you need such an amount of memory anyway (I assume you want several instances of that struct since you are then using containers)?

[Sheng]

AFAIK if you allocate a struct on heap ALL its field can only be allocated on heap...

Now some thoughts that may or may not be relevant :

• big structures like that should never ever be passed by value anywhere... allocate them on heap and use a QVector<myStruct*> instead of a QVector<myStruct> (or whatever container you may like more)
• Storing the data directly in the struct instead of a pointer to it is a HUGE design mistake IMO because it forces it to be allocated no matter what and that gets even worse when some copy has to be made. Using a pointer and allocating the data when it is needed would probably be a better idea, would allow sharing and would very probably cause a dramatic decrease of memory usage and a dramatic increase in speed
• huge contiguous arrays of memory are not a very wise design choice. Maybe you can split that data in smaller pieces? And do you need such an amount of memory anyway (I assume you want several instances of that struct since you are then using containers)?

Copy has to be made since I need to copy the image from the buffer to somewhere else, the buffer is updated quickly.

[lni]

AFAIK if you allocate a struct on heap ALL its field can only be allocated on heap...

FALSE.

Try this and tell me why it works.

#include <QVector>
#include <QString>
 
struct myStruc {
 QString myString;
 int *array;
 myStruc() {
    array = new int[ 3000*2000 ];
 }
 
};
 
void foo(const myStruc &v)
{
    printf("%d\n", v.array[1]);
}
 
int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);
 
    myStruc* currentStruc=new myStruc;
    foo(*currentStruc);
   QVector<myStruc> myVector;
   myVector.push_back(*currentStruc);
}

#include <QVector>
#include <QString>

struct myStruc {
 QString myString;
 int *array;
 myStruc() {
    array = new int[ 3000*2000 ];
 }

};

void foo(const myStruc &v)
{
    printf("%d\n", v.array[1]);
}

int main(int , char* [])
{
    printf("hello\n");
    fflush(stdout);
    sleep(1);

    myStruc* currentStruc=new myStruc;
    foo(*currentStruc);
   QVector<myStruc> myVector;
   myVector.push_back(*currentStruc);
}

To copy to clipboard, switch view to plain text mode 

[wysota]

FALSE.

Try this and tell me why it works.

:-)

Explain this:

#include "stdio.h"                                                                                                   
#include <QString>                                                                                                   
 
struct A {
 QString x;
 int array[1000];
};               
 
int main(){
   A a;    
   A *b = new A;
  printf("Addr of a:        0x%08x\n", (unsigned int)&a);
  printf("Addr of a.array:  0x%08x\n", (unsigned int)a.array);
  printf("Addr of b:        0x%08x\n", (unsigned int)b);
  printf("Addr of b->array: 0x%08x\n", (unsigned int)b->array);
  return 0;
}

#include "stdio.h"                                                                                                   
#include <QString>                                                                                                   

struct A {
 QString x;
 int array[1000];
};               

int main(){
   A a;    
   A *b = new A;
  printf("Addr of a:        0x%08x\n", (unsigned int)&a);
  printf("Addr of a.array:  0x%08x\n", (unsigned int)a.array);
  printf("Addr of b:        0x%08x\n", (unsigned int)b);
  printf("Addr of b->array: 0x%08x\n", (unsigned int)b->array);
  return 0;
}

To copy to clipboard, switch view to plain text mode 

Result:

Addr of a:        0xbfa5008c
Addr of a.array:  0xbfa50090
Addr of b:        0x089fe1b8
Addr of b->array: 0x089fe1bc

Addr of a:        0xbfa5008c
Addr of a.array:  0xbfa50090
Addr of b:        0x089fe1b8
Addr of b->array: 0x089fe1bc

To copy to clipboard, switch view to plain text mode 

As you see a and a.array are on the stack and b and b->array are on heap.

Your "example" works because even if "myStruc" is allocated on the stack, its array member will be allocated on the heap. But it doesn't mean that if myStruc is allocated on the heap, array is (or can be) allocated on the stack

[wysota]

By the way, push_back is not a problem. This crashes on the QVector constructor and I think I know why

#include <QVector>
#include <QString>
 
struct StructOfDeath {
  QString some;
  QString innocent;
  int variables;
  double And;
  int memberOfDeath[3000*2000];
  StructOfDeath(){ variables = 0; And = 1;}
};
 
int main(){
 
  QVector<StructOfDeath> vector;
  vector.reserve(30);
  for(int i=0;i<10;i++)
    vector.push_back(StructOfDeath());
  printf("Done\n");
  return 0;
}

#include <QVector>
#include <QString>

struct StructOfDeath {
  QString some;
  QString innocent;
  int variables;
  double And;
  int memberOfDeath[3000*2000];
  StructOfDeath(){ variables = 0; And = 1;}
};

int main(){

  QVector<StructOfDeath> vector;
  vector.reserve(30);
  for(int i=0;i<10;i++)
    vector.push_back(StructOfDeath());
  printf("Done\n");
  return 0;
}

To copy to clipboard, switch view to plain text mode 

[lni]

:-)

As you see a and a.array are on the stack and b and b->array are on heap.

Your "example" works because even if "myStruc" is allocated on the stack, its array member will be allocated on the heap. But it doesn't mean that if myStruc is allocated on the heap, array is (or can be) allocated on the stack

Do some google search on "array in stack overflow", or at
http://www.devx.com/tips/Tip/14276

[faldzip]

Do some google search on "array in stack overflow", or at
http://www.devx.com/tips/Tip/14276

I can't see any connection between that article and things you are saying :P
As for me it's simple:

If there's code like this:

A a;

A a;

To copy to clipboard, switch view to plain text mode 

object a of type A is created on stack. If that code appears as a local variable in some function/method then at the end of the function/method object a will be deleted from stack, because program is returning to the caller, to place where call appeared.
(that's why there's stack overflow in infinite recursion -> infinite storing the called method variables and no returns, which will remove these variables from stack)

But if there's code like this:

A *a = new A;

A *a = new A;

To copy to clipboard, switch view to plain text mode 

instead of previously shown, then it's different story :] it means that your local variable created on a stack is a pointer (32-bit address = 4B) which points to the object of type A created on heap - not afecting stack in any way.

So if there is an array in struct created dynamically on a heap:

struct myStruc {
 QString myString;
 int *array;
 myStruc() {
    array = new int[ 3000*2000 ];
 }
 
};

struct myStruc {
 QString myString;
 int *array;
 myStruc() {
    array = new int[ 3000*2000 ];
 }
 
};

To copy to clipboard, switch view to plain text mode 

then this struct size is just a size of myString and size of pointer to an array. An array itself is always on heap (beacause of operator new).


If there is any mistake in my understanding of stack/heap, feel free to provide any interesting links on this topic.

[wysota]

Do some google search on "array in stack overflow", or at
http://www.devx.com/tips/Tip/14276

Honestly I don't see any connection between the issue of the array being on the stack or not and the article. I think the example I gave you is a proof that can't be undermined.

Stack addresses have high values because in most (all?) modern architectures the stack expands "downwards" (to lower memory addresses) so there has to be enough space for the stack to expand until it reaches the heap space (that is (usually?) between the global variable address space and the stack address space). The example clearly shows that a.array is positioned 4 bytes away from the beginning of the structure in the same adress region so it has to be inside the struct as sizeof(A) will return a value larger than 4 bytes so a logical conclusion follows that both "objects" have to be positioned in the same "part" of memory. The same goes for the second variant, the difference is also 4 bytes and the size of the structure remains the same.

Here is a result of the extended version of my previous example. Note the difference in structure sizes and addresses.

Addr of a:        0xbfcaebb0
Addr of a.array:  0xbfcaebb4
Addr of b:        0x0808d1b8
Addr of b->array: 0x0808d1bc
Size of struct A: 4004
 
Addr of c:        0xbfcafb54
Addr of c.array:  0x0808e168
Addr of d:        0x0808f110
Addr of d->array: 0x0808f120
Size of struct B: 8

Addr of a:        0xbfcaebb0
Addr of a.array:  0xbfcaebb4
Addr of b:        0x0808d1b8
Addr of b->array: 0x0808d1bc
Size of struct A: 4004

Addr of c:        0xbfcafb54
Addr of c.array:  0x0808e168
Addr of d:        0x0808f110
Addr of d->array: 0x0808f120
Size of struct B: 8

To copy to clipboard, switch view to plain text mode 

And the code itself...

#include "stdio.h"
#include <QString>
 
struct A {
 QString x;
 int array[1000];
};
 
struct B {
  QString x;
  int *array;
  B(){ array = new int[1000]; }
};
 
int main(){
   A a;
   A *b = new A;
   B c;
   B *d = new B;
  printf("Addr of a:        0x%08x\n", (unsigned int)&a);
  printf("Addr of a.array:  0x%08x\n", (unsigned int)a.array);
  printf("Addr of b:        0x%08x\n", (unsigned int)b);
  printf("Addr of b->array: 0x%08x\n", (unsigned int)b->array);
  printf("Size of struct A: %d\n", sizeof(A));
  printf("\n");
  printf("Addr of c:        0x%08x\n", (unsigned int)&c);
  printf("Addr of c.array:  0x%08x\n", (unsigned int)c.array);
  printf("Addr of d:        0x%08x\n", (unsigned int)d);
  printf("Addr of d->array: 0x%08x\n", (unsigned int)d->array);
  printf("Size of struct B: %d\n", sizeof(B));
 
  return 0;
}

#include "stdio.h"
#include <QString>

struct A {
 QString x;
 int array[1000];
};

struct B {
  QString x;
  int *array;
  B(){ array = new int[1000]; }
};

int main(){
   A a;
   A *b = new A;
   B c;
   B *d = new B;
  printf("Addr of a:        0x%08x\n", (unsigned int)&a);
  printf("Addr of a.array:  0x%08x\n", (unsigned int)a.array);
  printf("Addr of b:        0x%08x\n", (unsigned int)b);
  printf("Addr of b->array: 0x%08x\n", (unsigned int)b->array);
  printf("Size of struct A: %d\n", sizeof(A));
  printf("\n");
  printf("Addr of c:        0x%08x\n", (unsigned int)&c);
  printf("Addr of c.array:  0x%08x\n", (unsigned int)c.array);
  printf("Addr of d:        0x%08x\n", (unsigned int)d);
  printf("Addr of d->array: 0x%08x\n", (unsigned int)d->array);
  printf("Size of struct B: %d\n", sizeof(B));

  return 0;
}

To copy to clipboard, switch view to plain text mode 

[Sheng]

Thank you all for your posts. I think I am clear of everything about Qt containers.
I also think every one learn something new, if not, you are good enough.

[wysota]

Sheng, a side question - why don't you use QImage or QByteArray instead of that huge array of yours?