QHash : Virtual memory not released

[wysota]

Defragmenting virtual memory is a nonsense idea anyway, even if you have a 4GB contiguous chunk of memory, theres nothing stopping the OS mapping those contiguous chunks to a random chunks of physical memory.

That's not exactly true. The idea of defragmenting memory is that most of memory of other processes gets pushed to swap and physical memory gets released. Then you can request a big chunk of memory that otherwise might be problematic due to swapping and the fact that memory is allocated in pages and not bytes. Consider the fact that you have 256 units of memory in total and page size is 16 units. If some process requests one unit of memory, 16 units get assigned to it and other processes can only request 240 units of memory without swapping. If you force the other process to be preemptied and its memory moved to swap, you can allocate 256 units of physical memory instead of 240 units of physical memory and 16 units of swap space. What you gain is that the swapping process is moved in time and you get a continuus chunk of physical memory with perspectives of faster access to physical memory. At least in theory. In practice there are other processes running and interacting with your process so it all becomes less predictable.

As for the original problem, find the border case where the allocation suceeds and make sure the hash is destroyed before that and that all values are correct (watch out for data type limitations).

[squidge]

Yes, the only real difference is when the delay occurs to swap data to disk - when you allocate memory for no reason to flush the system, or when you allocate the memory that you actually want. It doesn't make much (if any) difference to how much memory you can get.

I do agree however that sometimes you have a problem when the system is busy moving memory around that a memory allocation may fail (this is documented in the MSDN), but you shouldn't try to allocate huge amounts of memory in your app just to get everyone else to purge there buffers to swap space.

To an application, it has no idea whether a contiguous area of memory is physically contiguous or only logically contiguous, as each process has its own memory range which the OS chooses itself where to put in physical memory, and its only when you write to this memory does it actually get allocated (See below).

The other problem is that modern OSs use a "Copy on write" strategy, so they'll quite happily let you allocate a few GB of memory, but will not actually allocate it from the physicall memory store until you store something in it. This is why programs sometimes crash without warning - you write into another page of memory, the OS tries to allocate that page, finds out there no free physically memory left and will then attempt to move other memory to the swap file. If the swap file is full too, the process gets terminated unless its a critical system process, in which case it will pick a process and kill it using a certain algorithm (normally preferrring the big memory eaters)

Try it on Linux without a swap file, you'll find that a system with 512MB of RAM will happily let several processes allocate several GB of memory. Then write to that memory and watch the system kill you.

[wysota]

It doesn't make much (if any) difference to how much memory you can get.

It doesn't make difference how much virtual memory you can get. Things are different with physical memory (and on systems that have swap disabled it gets more significant as you can't use all the memory in the system).

but you shouldn't try to allocate huge amounts of memory in your app just to get everyone else to purge there buffers to swap space.

True but there are other things one shouldn't do or be able to do but Windows still does it (i.e. FindWindow/PostMessage to every possible window in the system).

[squidge]

True but there are other things one shouldn't do or be able to do but Windows still does it (i.e. FindWindow/PostMessage to every possible window in the system).

It seems in the latest version (Windows 7), security has been tightened up quite a bit, from the docs it seems that you can no longer post to another applications message loop. You have to request such functionality first now, both in your manifest file and via the API, which causes the UAC to prompt the user.

But thats not going to help the OP with there memory problem. Maybe they can run the application under OllyDbg, that way you can see the entire memory map for the process and see exactly where everything is being put. To use it properly however, it does help to have some knowledge of the x86 instruction set, as thats its primary output mode.

[wysota]

I'm still waiting for him to detect the border case. I doubt the hash has anything to do with it.

[squidge]

I agree, once the hash has been freed, the memory should be pretty much the same as before.