Is there a maximum vector length other than a memory limit?

[tbscope]

What are your stack and heap sizes?

[SixDegrees]

I'm not a Windows guy, but the failure of even Plain Old C Arrays to work suggests some sort of OS limit imposed on your processes. In Unix, both the OS and the user can fiddle with the maximum memory size a given process can allocate; this is normally set to "unlimited," but I've seen it restricted on some systems. I suspect Windows is probably "helping" you by imposing such a limit. No idea what the command would be to increase it, though.

Here's an article on the subject - sounds like this might be worth investigating.

[agerlach]

I'm not a Windows guy, but the failure of even Plain Old C Arrays to work suggests some sort of OS limit imposed on your processes. In Unix, both the OS and the user can fiddle with the maximum memory size a given process can allocate; this is normally set to "unlimited," but I've seen it restricted on some systems. I suspect Windows is probably "helping" you by imposing such a limit. No idea what the command would be to increase it, though.

Here's an article on the subject - sounds like this might be worth investigating.

You are on to something, it looks like my program is compiling at a 32 bit application which is limited to 2 GB of RAM. I created an empty project in VS2010 and compile as x64 and I was able to allocate significantly more memory. I was using Qt creator for development but I don't really know how to set it up to build my projects as x64. Would it be easier for me to just start using VS instead?

[SixDegrees]

Again, I'm not Windows-savvy, but make sure your QMAKESPEC is pointing to the 64-bit version of its files, not the 32-bit version.

[ChrisW67]

As I understand it, the bundled MingW environment in the Qt SDK is 32-bit only.

[agerlach]

OK, for my project I am using Qt, VTK, and the newmat C++ matrix library. I compiled newmat to x64 and downloaded the source for Qt and ran configure.exe -> nmake -> nmake install from the VS2008 x64 command prompt. Everything built fine. I then built VTK directed at the x64 Qt build in x64. Again, everything built fine. I guess my question is on how to start using the x64 version of Qt I built in Qt Creator and how do I handle QMAKESPEC. I don't see anything in the mkspecs folder to seems appropriate. There is only win32 for msvc2008.

[JohannesMunk]

The problem is that the vector requires all that data to be in one big chunk.

Your correlation will probably be interframe for each pixel? So that you need fast access to different frames of the same pixel.

Why don't you manage your data as a 256 long QList of 714838 double vectors?

Access won't be slower. But you allow for some fragmentation.

Joh

[agerlach]

The problem is that the vector requires all that data to be in one big chunk.

Your correlation will probably be interframe for each pixel? So that you need fast access to different frames of the same pixel.

Why don't you manage your data as a 256 long QList of 714838 double vectors?

Access won't be slower. But you allow for some fragmentation.

Thanks, that makes sense, but what do you mean by "interfram for each pixel". The following equation is how the correlation is calculated. P is the randomly selected image and Q is one of the 714838 images in the training stack. pi and qi are the ith pixel values of P or Q respectively. Their are 256 pixels per image. Normally W^2 = 256, but in this case it equals the number of pixels that are nonzero in both P and Q.
l Correlation..png

[JohannesMunk]

Ah ok. So you correlate whole frames. And you need to correlate every new frame against your complete training?

Then you should definitely store the individual pixel and squarepixels sums to speed up calculation of the denominator of your formula.

Store a struct/class for each frame with the rawpixel data and the precalculated values.

Put all your frames into a QList. That will effectively be an array of pointers to your structs/classes. As that array will be quite small and each frame-struct is small, you should not have a problem allocating that memory.
As you will need to resolve those only once for each much more expensive correlation-calculation, that won't have an impact on performance, either.

HIH

Johannes

[agerlach]

Ah ok. So you correlate whole frames. And you need to correlate every new frame against your complete training?

Then you should definitely store the individual pixel and squarepixels sums to speed up calculation of the denominator of your formula.

Store a struct/class for each frame with the rawpixel data and the precalculated values.

Put all your frames into a QList. That will effectively be an array of pointers to your structs/classes. As that array will be quite small and each frame-struct is small, you should not have a problem allocating that memory.
As you will need to resolve those only once for each much more expensive correlation-calculation, that won't have an impact on performance, either.

Just to make sure I understand everything, when you say "... as that array will be quit small" you are refering to its size in memory since it is just a array of pointers? If so, this make a ton of sense and is a huge help. Thanks!

On a second note, I never thought about pre calculating the sums in the equation. I never did in the past because only the pixel values in Q that correspond to non zero pixel values in P are used in the calculation. So if Q = {1 2 3} and P = {0 0 1} then sum(Q) = 3 since Q(0) and Q(1) are not used. But now that I think about it since the images are never all zero it would be more efficient to calculate the sum and then subtract the elements that correspond to a 0 value in P.

[JohannesMunk]

Just to make sure I understand everything, when you say "... as that array will be quit small" you are refering to its size in memory since it is just a array of pointers? If so, this make a ton of sense and is a huge help. Thanks!

Thats exactly what it means! Of course the overall memory usage will be equally big, but now the memory manager can decide where to put all the little pieces(=frames), and is not forced (and doesn't fail) to allocate a big continuous chunk.

On a second note, I never thought about pre calculating the sums in the equation. I never did in the past because only the pixel values in Q that correspond to non zero pixel values in P are used in the calculation. So if Q = {1 2 3} and P = {0 0 1} then sum(Q) = 3 since Q(0) and Q(1) are not used. But now that I think about it since the images are never all zero it would be more efficient to calculate the sum and then subtract the elements that correspond to a 0 value in P.

If Q={1,2,3} then sum(Qi)=6. if P={0,0,1} sum(Pi*Qi)=3. I was only talking of the denominator of your formula. There are no mixed (p and q) sum expressions there.

Johannes

[agerlach]

If Q={1,2,3} then sum(Qi)=6. if P={0,0,1} sum(Pi*Qi)=3. I was only talking of the denominator of your formula. There are no mixed (p and q) sum expressions there.

No, thats not correct. The calculation is only performed using pixels in Q and P that whose values are non zero in BOTH P and Q. Imagine you have a for loop before the correlation calculation that modifies Q based on P. Say Q = {1,2,3} and P = {0,0,1} with the following for loop:

for(ii=0; ii < 3; ii++) {
if(P[ii] == 0)
Q[ii]=0;
elseif(Q[ii] == 0)
P[ii]=0;

for(ii=0; ii < 3; ii++) {
if(P[ii] == 0)
Q[ii]=0;
elseif(Q[ii] == 0)
P[ii]=0;

To copy to clipboard, switch view to plain text mode 

then Q = {0,0,3} and sum(Qi) = 3 not 6.

Then the equation is used, but since you don't know P when the training stack is created you cannot precalculate the sums. My point was that time required to calculate the correlation is important not the time to calculate the training set. So if I precalculate the sums when I generate the training set and then subtract the values at Q[ii] when P[ii]==0 when calculating the correlation I would reduce the computation in calculating the correlation although I increased the total number of calculations.

[agerlach]

OK, for my project I am using Qt, VTK, and the newmat C++ matrix library. I compiled newmat to x64 and downloaded the source for Qt and ran configure.exe -> nmake -> nmake install from the VS2008 x64 command prompt. Everything built fine. I then built VTK directed at the x64 Qt build in x64. Again, everything built fine. I guess my question is on how to start using the x64 version of Qt I built in Qt Creator and how do I handle QMAKESPEC. I don't see anything in the mkspecs folder to seems appropriate. There is only win32 for msvc2008.

Apparently Qt Creator handles the QMAKESPEC for you based on the build of qmake you point it to. Everything works good except debugging. I download cdb for x64 and pointed Qt Creator to it but it has the following error:

Unable to load the debugger engine library 'C:\Program Files \Debuggint Tools for Windows (x64)\dbghelp.dll': Cannot load library C:\Program Files\Debugging Tools for Windows (x64)\dbghelp.dll

even though these files exist at that location.

[agerlach]

What are your stack and heap sizes?

I'm sorry, but could you tell me how to check that?