It seems that Qt does not release unused memory

[iw2nhl]

you can find memory leaks with this very handy tool:
valgrind (http://valgrind.org/)

Thanks for the suggestion, but I already used it!
See my post #5.
No memory leaks are shown.
Just a few KB caused by Qt and other linked libraries.

[iw2nhl]

I decided to make a very simple test.
It allocates a QVector, then it shows a message so that you can test memory usage, then the vector is deleted and you can test again memory usage.
In my system, both VmRss and VmSize do not change (actually just some KB).
Please, tell me if I'm doing something wrong!

#include <QtGui>
 
int main(int argc, char *argv[]) {
    QApplication app(argc, argv);
 
    // Memory allocation
    QVector<double> *vector = new QVector<double>;
    vector->resize(10000);
    QMessageBox::information(0, "Memory Allocation Test", "Memory allocated");
 
    // Memory deletion
    vector->clear();
    vector->squeeze();
    delete vector;
    QMessageBox::information(0, "Memory Allocation Test", "Memory deleted");
 
    return 0;
}

#include <QtGui>

int main(int argc, char *argv[]) {
    QApplication app(argc, argv);

    // Memory allocation
    QVector<double> *vector = new QVector<double>;
    vector->resize(10000);
    QMessageBox::information(0, "Memory Allocation Test", "Memory allocated");

    // Memory deletion
    vector->clear();
    vector->squeeze();
    delete vector;
    QMessageBox::information(0, "Memory Allocation Test", "Memory deleted");

    return 0;
}

To copy to clipboard, switch view to plain text mode 

[iw2nhl]

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.

double *vector = new double[10000];
delete[] vector;

double *vector = new double[10000];
delete[] vector;

To copy to clipboard, switch view to plain text mode 

[wysota]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.

[marcel]

On linux, what "ps" or "top" have to say?
Have you tested with other tools at all?

What if after closing the files you open an application that takes a lot of memory? Does vmrss drops in any way?
The vmsize is explainable since the virtual memory manager won't adjust the swap every time a process deallocates some memory.

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.

I really doubt that a bug of this magnitude could exist in the memory manager.

[wysota]

An hour ago I implemented a test program that was allocating 15000 QVector<int> objects, filling them with data and reporting the memory usage through reading /proc/pidnum/status, then deallocating them (by deleting the vectors) and repeating the procedure a few times. I can't show you the sources as running the application rendered my computer unresponsive and I had to make a hard reboot (and the program was stored in /tmp). Nevertheless I monitored the memory usage and it was not what I expected. VmRSS (physical memory) stayed the same during subsequent iterations (which was expected), but VmSize (virtual memory) kept rising during each iteration (which wasn't expected) until it ate all the swap space and froze the system solid. All that means one of two (or more?) things - either I made some stupid mistake that caused memory not to be released properly or there is something wrong with memory management either in Qt or the kernel or.... I don't know where (compiler?)... Or I don't know what VmSize is...
The app looked essentially like this (+reporting mem usage everywhere):

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

To copy to clipboard, switch view to plain text mode 

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer. The funny thing is I have 1,2GB swap + 768MB physical memory in my machine which is less than VmSize was reporting as reserved for the process before the system started freezing.

[Gopala Krishna]

An hour ago I implemented [..]

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

To copy to clipboard, switch view to plain text mode 

I couldn't understand much on how you extract the info of memory usage but atleast i found this code faulty.
You initially allocate 15k QVector<int> objects(with 10k int each) and push it to "vecs" vector and you just "delete" the 15k vectors in first pass. Note you aren't clearing the pointers in the "vect" vector.
In second pass you append 15k QVector<int> objects i.e now there are 30k pointers in vecs and you delete the first 15k "invalid" pointers which seems wrong.

Probably you could have simplified code like this. (can you try it ? )

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    qDeleteAll(vecs);
    //You missed this clear statement
    vecs.clear();
 }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    qDeleteAll(vecs);
    //You missed this clear statement
    vecs.clear();
 }
}

To copy to clipboard, switch view to plain text mode 

Of course you might have just done a cut and paste mistake !

[wysota]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

[Gopala Krishna]

qDeleteAll(vecs) is equivalent to for(...) delete vecs[i].

I know
I just wanted to say you missed vecs.clear();

[wysota]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

[Gopala Krishna]

Oh, you meant clear()... No, I didn't miss it. The vector goes out of scope and so it gets cleared anyway. But even if I did miss clear, it would only make my app hold an additional of 45000*sizeof(int*) = ~180kB of memory.

Sorry , but have a look at the code again. vecs is declared before the outermost for loop so it wont get cleared.
I am not sure how it relates here but deleting an invalid pointer does result in unexpected behavior.

[wysota]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer.

[Gopala Krishna]

Yes, I understand your point. But note that I wrote this wasn't the actual code. In the real app I was probably using the index operator to fill the vector, otherwise delete vecs[i] would have crashed the application during the second iteration of the loop (because of trying to delete an unallocated block of memory). And if you read my original post again, you'll see this line:

That is the reason i put this is in one of the post above

Of course you might have just done a cut and paste mistake !

Anyways back to topic

[iw2nhl]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.

Ok, I made a too simple test.
Changing size from 10000 to 1000000 worked as espected. The memory has been allocated and deallocated.
I'll try to make a better test code! Really sorry for my mistake .

Have you tested with other tools at all?

Yes, I tried using directly the /proc dirs.
# cat /proc/pid_num/status

Summing up a simple test works as expected, but my program does not (but only with Linux, with Windows works as expected).
I'm trying to enable/disable some portions of my code to find where the problem arises.

[marcel]

Or I don't know what VmSize is...

It is process swap size + the in-use physical memory.

Regards