It seems that Qt does not release unused memory

[jacek]

How do you measure the memory used by your application?

[iw2nhl]

How do you measure the memory used by your application?

I use KDE System Guard 1.2.0, part of KDE 3.5.7: in the process table I read the columns "VmSize" and "VmRss".
While on Windows 2000 I use "Task Manager", with "VM Size" and "Mem Usage" columns.

I noticed a difference between Linux and Windows: Linux never releases the memory, while Windows releases most of it, but not all.
Eg.:
If the program, just started takes 10 MB of RAM and I load data for 30 MB (total 40 MB), when I unload the data, the program occupies 12 MB. From this point on, when I load and unload a file, it returns always to 12 MB (and never to 10 MB).
On Linux instead it remains always at the peak memory usage reached (in this example 40 MB), irrespective of what I load/unload.

[aMan]

How do you save your data?
In a QList? Do you delete the container or just empty it?

Maybe Qt doesn't frees the memory of the container but keeps it as a buffer.

[wysota]

On Linux instead it remains always at the peak memory usage reached (in this example 40 MB), irrespective of what I load/unload.

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory. Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

[iw2nhl]

Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

This is exactly what is happening.

I could try to use all available memory with another program and see if the memory footprint is reduced.
I'll try it asap!
Thanks

[iw2nhl]

Ok, I loaded files up to use 304 MB of VmSize (about 270-280 MB of VmRss), then I closed all of them and I started opening every sort of program.
The result?
Memory usage of the program was identical for VmSize, while VmRss diminished about of 20-30 MB.
More than 130 MB of Swap memory were used (it was never happened before).

I have 512 MB of RAM in my system and a Swap partition of 258 MB.

I don't know how to interpret the result. Is this a behaviour of the kernel? Is this a Qt bug? Is there something wrong in my code?
I really don't know

[wysota]

In your case VmSize can go up to over 700MB. I think VmRss is the amount of memory really occupied by the application.

http://www.mozilla.org/projects/foot...int-guide.html

BTW. Increase your swap space. It should be about twice as large as the amount of RAM you have in your machine.

[niko]

you can find memory leaks with this very handy tool:
valgrind (http://valgrind.org/)

its fairly easy to use....


niko

[iw2nhl]

you can find memory leaks with this very handy tool:
valgrind (http://valgrind.org/)

Thanks for the suggestion, but I already used it!
See my post #5.
No memory leaks are shown.
Just a few KB caused by Qt and other linked libraries.

[iw2nhl]

I decided to make a very simple test.
It allocates a QVector, then it shows a message so that you can test memory usage, then the vector is deleted and you can test again memory usage.
In my system, both VmRss and VmSize do not change (actually just some KB).
Please, tell me if I'm doing something wrong!

#include <QtGui>
 
int main(int argc, char *argv[]) {
    QApplication app(argc, argv);
 
    // Memory allocation
    QVector<double> *vector = new QVector<double>;
    vector->resize(10000);
    QMessageBox::information(0, "Memory Allocation Test", "Memory allocated");
 
    // Memory deletion
    vector->clear();
    vector->squeeze();
    delete vector;
    QMessageBox::information(0, "Memory Allocation Test", "Memory deleted");
 
    return 0;
}

#include <QtGui>

int main(int argc, char *argv[]) {
    QApplication app(argc, argv);

    // Memory allocation
    QVector<double> *vector = new QVector<double>;
    vector->resize(10000);
    QMessageBox::information(0, "Memory Allocation Test", "Memory allocated");

    // Memory deletion
    vector->clear();
    vector->squeeze();
    delete vector;
    QMessageBox::information(0, "Memory Allocation Test", "Memory deleted");

    return 0;
}

To copy to clipboard, switch view to plain text mode 

[iw2nhl]

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.

double *vector = new double[10000];
delete[] vector;

double *vector = new double[10000];
delete[] vector;

To copy to clipboard, switch view to plain text mode 

[wysota]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.

[marcel]

On linux, what "ps" or "top" have to say?
Have you tested with other tools at all?

What if after closing the files you open an application that takes a lot of memory? Does vmrss drops in any way?
The vmsize is explainable since the virtual memory manager won't adjust the swap every time a process deallocates some memory.

I tested also with a standard vector and it has the same behaviour, so probably it is a kernel bug/feature.

I really doubt that a bug of this magnitude could exist in the memory manager.

[wysota]

An hour ago I implemented a test program that was allocating 15000 QVector<int> objects, filling them with data and reporting the memory usage through reading /proc/pidnum/status, then deallocating them (by deleting the vectors) and repeating the procedure a few times. I can't show you the sources as running the application rendered my computer unresponsive and I had to make a hard reboot (and the program was stored in /tmp). Nevertheless I monitored the memory usage and it was not what I expected. VmRSS (physical memory) stayed the same during subsequent iterations (which was expected), but VmSize (virtual memory) kept rising during each iteration (which wasn't expected) until it ate all the swap space and froze the system solid. All that means one of two (or more?) things - either I made some stupid mistake that caused memory not to be released properly or there is something wrong with memory management either in Qt or the kernel or.... I don't know where (compiler?)... Or I don't know what VmSize is...
The app looked essentially like this (+reporting mem usage everywhere):

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

int main(){
  QVector<QVector<int>*> vecs;
  for(int r=0;r<3;r++){
    for(int i=0;i<15000;i++){
      QVector<int> *v = new QVector<int>;
      for(int j=0;j<10240;j++)
        (*v) << 7;
      vecs << v; // or was it vecs[i] = *v; ?
    }
    for(int i=0;i<15000;i++){
      delete vecs[i];
    }
  }
}

To copy to clipboard, switch view to plain text mode 

A probable error is not squeezing "vecs" after each iteration, but it shouldn't make it occupy over 2GB of ram (3*15000*sizeof(int) = ~180kB) - at worst it should kill the process on trying to delete an invalid pointer. The funny thing is I have 1,2GB swap + 768MB physical memory in my machine which is less than VmSize was reporting as reserved for the process before the system started freezing.

[iw2nhl]

10000*sizeof(double) = ~800kB

BTW. Nobody said resize() actually causes any allocation... Try assigning a value to one of the last elements.

Ok, I made a too simple test.
Changing size from 10000 to 1000000 worked as espected. The memory has been allocated and deallocated.
I'll try to make a better test code! Really sorry for my mistake .

Have you tested with other tools at all?

Yes, I tried using directly the /proc dirs.
# cat /proc/pid_num/status

Summing up a simple test works as expected, but my program does not (but only with Linux, with Windows works as expected).
I'm trying to enable/disable some portions of my code to find where the problem arises.

[marcel]

Or I don't know what VmSize is...

It is process swap size + the in-use physical memory.

Regards

[Brandybuck]

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory.

This is true for Linux, but other Unix systems have differing memory allocation behavior. Try your application under FreeBSD or Solaris for a comparison.

[wysota]

Yes, of course. That's natural as every system uses a different allocator implementation. The author said he was running Linux. Furthermore one Linux based system may behave differently than another, it depends on the kernel it is running (both version and options).

[fluxar]

As far as I know this is a normal behaviour. Your kernel prefers to have the block of memory assigned to a given process in case it wants more memory at some later point in time. If you ran out of memory, it'd be freed and assigned to the process being in need of that memory. Try opening a large file, processing it and then repeating with a smaller file. You'll probably notice the memory footprint remains the same, which means the memory for the smaller file was allocated from within the previously deallocated chunk.

I also thought I had a memleak, but after I tested this (loaded a bunch of other applications to fill the memory) my application freed the memory that it no longer used. Thanks wysota!