Is there a maximum vector length other than a memory limit?

[agerlach]

I am using Win 7 64 bit with 12 GB RAM.
I want to allocate a QVector<double> of length 714838*256 by:

QVector<double> *qwert = new QVector<double>(714838*256);

QVector<double> *qwert = new QVector<double>(714838*256);

To copy to clipboard, switch view to plain text mode 

but I get a bad_alloc() exception. So I tried it with just:

double * qwert = new double[714838 * 256];

double * qwert = new double[714838 * 256];

To copy to clipboard, switch view to plain text mode 

And I still get an error. After a few trials I found that I can allocate 178077685 doubles but not 178077686. But,

QVector<double> *qwert = new  QVector<double>(178077685);

QVector<double> *qwert = new  QVector<double>(178077685);

To copy to clipboard, switch view to plain text mode 

still throws the same exception. I show this to be ~1.32 GB of doubles wihich is << 12 GB or RAM available.
I've looked all day online to find a reason why I cannot allocate a double array any larger and the only explanation I can find is a RAM limitation, which is not the case here.

[tbscope]

There are more limits on your pc besides the amount of ram. (as in, do you know exactly where this memory is reserved?)

Why, dear oh dear, why?
Why do you need that much memory?

To me this screams "Complete and utter bad design"
Can you tell why you need to do this?

[SixDegrees]

What happens if you try to allocate a simple array of doubles, as in

double* t = new double[178077686];

double* t = new double[178077686];

To copy to clipboard, switch view to plain text mode 

Are the results different?

[agerlach]

What happens if you try to allocate a simple array of doubles, as in

double* t = new double[178077686];

double* t = new double[178077686];

To copy to clipboard, switch view to plain text mode 

Are the results different?

I may have been unclear but I tried that. This works:

double* t = new double[178077685];

double* t = new double[178077685];

To copy to clipboard, switch view to plain text mode 

but this does not:

double* t = new double[178077686];

double* t = new double[178077686];

To copy to clipboard, switch view to plain text mode 

[agerlach]

There are more limits on your pc besides the amount of ram. (as in, do you know exactly where this memory is reserved?)

Why, dear oh dear, why?
Why do you need that much memory?

To me this screams "Complete and utter bad design"
Can you tell why you need to do this?

The 714838*256 size represents 714838 16x16 "images" called a training stack. This is used by taking another 16x16 "image" and correlating it with all the images in the stack. This needs to be performed fast. I currently have this algorithm built in Matlab and I have no problem allocating this much memory. Based on the time it takes to do this in Matlab and based on the speed ups I have experienced in the past from moving to C++ I expect to do this in < 1 s. Due to the speed requirements I would rather not have to write intermediate steps to the hard disk. Plus this code will not be deployed so I can control any hardware constraints.

If anybody has any suggestions on betters ways to do this that will not sacrifice speed, I am open to ideas.

What other limits would I have other than RAM? And what do you mean by "do you know exactly where this memory is reserved?"

[tbscope]

What other limits would I have other than RAM? And what do you mean by "do you know exactly where this memory is reserved?"

What sits between your physical ram and your program code?
It's a chain with limits.

For example, you will not be able to use the 12GB of ram on a 32 bit system

[agerlach]

I understand that I will not be able to use 12GB RAM on a 32 bit system, but this is on a 64 bit OS.

[tbscope]

What are your stack and heap sizes?

[SixDegrees]

I'm not a Windows guy, but the failure of even Plain Old C Arrays to work suggests some sort of OS limit imposed on your processes. In Unix, both the OS and the user can fiddle with the maximum memory size a given process can allocate; this is normally set to "unlimited," but I've seen it restricted on some systems. I suspect Windows is probably "helping" you by imposing such a limit. No idea what the command would be to increase it, though.

Here's an article on the subject - sounds like this might be worth investigating.

[agerlach]

I'm not a Windows guy, but the failure of even Plain Old C Arrays to work suggests some sort of OS limit imposed on your processes. In Unix, both the OS and the user can fiddle with the maximum memory size a given process can allocate; this is normally set to "unlimited," but I've seen it restricted on some systems. I suspect Windows is probably "helping" you by imposing such a limit. No idea what the command would be to increase it, though.

Here's an article on the subject - sounds like this might be worth investigating.

You are on to something, it looks like my program is compiling at a 32 bit application which is limited to 2 GB of RAM. I created an empty project in VS2010 and compile as x64 and I was able to allocate significantly more memory. I was using Qt creator for development but I don't really know how to set it up to build my projects as x64. Would it be easier for me to just start using VS instead?

[SixDegrees]

Again, I'm not Windows-savvy, but make sure your QMAKESPEC is pointing to the 64-bit version of its files, not the 32-bit version.

[ChrisW67]

As I understand it, the bundled MingW environment in the Qt SDK is 32-bit only.

[agerlach]

OK, for my project I am using Qt, VTK, and the newmat C++ matrix library. I compiled newmat to x64 and downloaded the source for Qt and ran configure.exe -> nmake -> nmake install from the VS2008 x64 command prompt. Everything built fine. I then built VTK directed at the x64 Qt build in x64. Again, everything built fine. I guess my question is on how to start using the x64 version of Qt I built in Qt Creator and how do I handle QMAKESPEC. I don't see anything in the mkspecs folder to seems appropriate. There is only win32 for msvc2008.

[agerlach]

What are your stack and heap sizes?

I'm sorry, but could you tell me how to check that?