Results 1 to 17 of 17

Thread: Memory allocation failure and crash in QVector

  1. #1
    Join Date
    Oct 2007
    Posts
    6
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Memory allocation failure and crash in QVector

    I found the crash when I used QVector. Actually it raises a question how to handle memory allocation failures gracefully with Qt container classes like QVector and etc.

    The code below demonstrates problem. To compile and run it I used Qt 4.3.2 and GCC/G++ 3.4.2 on Windows Server 2003. The STL version of vector correctly raises exception but QVector just generates segmentation fault.

    try
    {
    std::vector<qlonglong>* pVector =
    new std::vector<qlonglong>(INT_MAX - 0x1000);
    }
    catch(...)
    {
    }

    try
    {
    QVector<qlonglong>* pQVector =
    new QVector<qlonglong>(INT_MAX - 0x1000);
    }
    catch(...)
    {
    }

    Can somebody shed light on this topic.

  2. #2
    Join Date
    Jan 2006
    Location
    Ohio
    Posts
    332
    Thanks
    37
    Thanked 8 Times in 4 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    Why do you want to create a pointer to a QVector? Why not just create a QVector and pass the size in to allocate and reserve the space for the QVector?

    Also, if you know ahead of time what size you want, why not just use an array and malloc the memory for it, as it is faster than a vector or QVector because of less overhead?

    As far as the syntax, it looks correct to me unless you just can't create a pointer to a QVector, which for some reason I doubt is the problem.

  3. #3
    Join Date
    Feb 2006
    Location
    Romania
    Posts
    2,744
    Thanks
    8
    Thanked 541 Times in 521 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    A memory request of that size will always fail on a regular computer...
    I am not sure, but QVector should assert when the allocation fails.
    Are you sure it is a seg fault and not an assert?

  4. #4
    Join Date
    Oct 2007
    Posts
    6
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    1. I especially selected such size to demonstrate the problem. But actually it can happen with any size under low memory conditions in OS.
    2. Will assertion help in release mode? I doubt.

    After short debug session I found that it crashes right in QVector's constructor:

    template <typename T>
    inline QVectorData *QVector<T>::malloc(int aalloc)
    {
    return static_cast<QVectorData *>(qMalloc(sizeof(Data) + (aalloc - 1) * sizeof(T)));
    }

    template <typename T>
    QVector<T>::QVector(int asize)
    {
    p = malloc(asize);
    d->ref.init(1);
    d->alloc = d->size = asize;
    d->sharable = true;
    d->capacity = false;
    if (QTypeInfo<T>::isComplex) {
    T* b = d->array;
    T* i = d->array + d->size;
    while (i != b)
    new (--i) T;
    } else {
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    qMemSet(d->array, 0, asize * sizeof(T)); // place where crash actually occurs
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    }
    }

  5. #5
    Join Date
    Oct 2007
    Posts
    6
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    Quote Originally Posted by ToddAtWSU View Post
    Why do you want to create a pointer to a QVector? Why not just create a QVector and pass the size in to allocate and reserve the space for the QVector?

    Also, if you know ahead of time what size you want, why not just use an array and malloc the memory for it, as it is faster than a vector or QVector because of less overhead?

    As far as the syntax, it looks correct to me unless you just can't create a pointer to a QVector, which for some reason I doubt is the problem.
    1. I just would like to use all power of QVector class but bug stops me from that. I posted the question just to find some possible workaround if problem is known already.
    2. Why you are thinking that pointer to arbitrary instance of C++ class is illegal?

  6. #6
    Join Date
    Oct 2007
    Posts
    6
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    Running code below I got another crash but now for QList:
    Qt Code:
    1. #define X86_PAGE_SIZE 4096
    2.  
    3. class MemChunk
    4. {
    5. char m_aChars[0x100 * X86_PAGE_SIZE];
    6.  
    7. public:
    8. MemChunk() { qMemSet(&m_aChars, 0, sizeof(m_aChars)); }
    9. };
    10.  
    11. void testQtList(void)
    12. {
    13. QList<MemChunk> list;
    14. MemChunk memChunk;
    15. while (true)
    16. {
    17. int before = list.size();
    18. try
    19. {
    20. list.append(memChunk);
    21. }
    22. catch(...)
    23. {
    24. qDebug() << "Exception";
    25. break;
    26. }
    27.  
    28. if (before == list.size())
    29. {
    30. qDebug() << "Break";
    31. break;
    32. }
    33. }
    34. }
    To copy to clipboard, switch view to plain text mode 


    (gdb) r
    Starting program: D:\Work\TestQt/debug\testQt.exe

    Program received signal SIGSEGV, Segmentation fault.
    0x004077c3 in probe ()
    Current language: auto; currently c++
    (gdb) bt
    #0 0x004077c3 in probe ()
    #1 0x1018f090 in QListData::shared_null () at painting/qdrawhelper.cpp:2227
    #2 0x0040162f in testQtList() () at testQt.cpp:39
    #3 0x0040145b in qMain(int, char**) (argc=1, argv=0x3d4c78) at testQt.cpp:11
    #4 0x00401edf in WinMain (instance=0x400000, prevInstance=0x0, cmdShow=10)
    at qtmain_win.cpp:120
    #5 0x0040197a in main ()
    (gdb)

    Now it happens append method of QList.

    My question now: Is Qt library able to handle memory allocation failures gracefully at all?

  7. #7
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Memory allocation failure and crash in QVector

    From what I see in the above code, Qt uses malloc() to allocate data (due to the later use of realloc() to move the data fast, probably) and this is a C call which, surprise, surprise, doesn't throw exceptions. Personally if you really used the above "test code", I'd say you need to redesign your approach. Allocating 16GB of data will ALWAYS fail on 32bit architecture, as you can't address more than 4GB of memory at once and having a dynamic vector of 2^30 items is a sick idea - try traversing it. You'd use a dynamic vector starting with small size and expand it when needed. In that case realloc() will work correctly and won't let you expand the vector more than OS allows.

    So you have two choices - using the slow but "exception-enabled" std::vector with broken code, using "broken" (when it comes to throwing exceptions) but very fast QVector with correct code or using a constant size array allocated using new[] and catch the exception yourself. If you really need to allocate such a large array (be it 2^30 or 2^20 items), I suggest you use a regular array or reconsider your design.

  8. #8
    Join Date
    Oct 2007
    Posts
    6
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    I would like precise again - I selected so big size only to demonstrate problem.
    Suppose that some application consumes a lot of memory and as result generates low memory condition. When I run my application then, at some point, even small allocation of memory by QVector, QList can trigger crash of application that is, at least, unprofessional and can lead to user’s data corruption that is more important.

    To demonstrate this problem please have a look into code below:
    Qt Code:
    1. #define X86_PAGE_SIZE 0x1000
    2.  
    3. void TestQt(void)
    4. {
    5. QVector<qlonglong> vector;
    6. MemoryConsumer();
    7. try
    8. {
    9. vector.resize(X86_PAGE_SIZE);
    10. }
    11. catch(...)
    12. {
    13. qDebug() << "Exception";
    14. }
    15. }
    16.  
    17. void MemoryConsumer(void)
    18. {
    19. int allocationSize = 0x100* X86_PAGE_SIZE;
    20.  
    21. while (true)
    22. {
    23. try
    24. {
    25. char* pChar = new char[allocationSize]; // especially made memory leak for simplicity
    26. }
    27. catch(...)
    28. {
    29. allocationSize /= 2;
    30. if (!allocationSize)
    31. {
    32. break;
    33. }
    34. }
    35. }
    36. }
    To copy to clipboard, switch view to plain text mode 

    It crashes right in resize() method that you recommended.

    As far as I know in C++ exist only two approaches for error handling:
    1. Error driven model when you analyzing returned error codes.
    2. Exception driven model when you expecting exceptions, catching them and providing some response to them



    The handling of memory allocation failures is also error handling. Without error handling it does not possible to create reliable and predictable application. Is not it?

    The second way is not possible with Qt since Trolltech developers decided to not use exceptions.
    But first way also impossible because of two points:
    1. It crashes in arbitrary places inside Qt library code.
    2. Many functions that allocate memory don’t return any return codes at all just void type.

    Do Trolltech tests own library with memory allocation failures? It seems to me that not.
    Trolltech just uses 3d approach of error handling, i.e. they just “forgot” about it at all.

    Now important question: Do they use this 3d approach for whole set Qt classes or only for Qt container classes?
    Actually I don’t have time now to test all Qt classes (including Qt GUI classes).

    Is it possible to configure Qt in way when it will handle memory allocations gracefully?

  9. #9
    Join Date
    Oct 2006
    Posts
    279
    Thanks
    6
    Thanked 40 Times in 39 Posts
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    You are right, this is a problem. I am afraid that Qt is still not exception safe. I guess this was imposed at some point by compiler limitations and bad stl implementations, but now this is getting to be a serious flaw. I hope that TT will rework the library at some point (Qt 5?), but that would be a major project. Exception safety doesn't come cheap and I know few libraries apart from stl, boost and the like who strive for it.

    In practice, though I wouldn't worry that much unless you are programming medical software, or something to do with aviation. In my experience it is very rare that an application throws a bad_alloc exception. Long before that, the virtual memory has started swapping like mad, making it orders of magnitude slower (i.e. unusable) without "officially" malfunctioning.

  10. #10
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Memory allocation failure and crash in QVector

    As I said - if they used new instead of malloc(), they could not have used realloc() to reallocate memory and as far as I know there is no equivalent to realloc() in C++. So the answer would be to use malloc() and check the return pointer for null, but then I'm not sure all the platforms respect returning null in case of a failed malloc. And if they don't, there is no sense checking the pointer (and maybe throwing the exception manually from within qMalloc), because malloc will abort the application anyway. I'll play a bit with QVector and get back to you when I have some results.

  11. #11
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Memory allocation failure and crash in QVector

    Ok, I made a test using the following application:
    Qt Code:
    1. #include <QVector>
    2.  
    3.  
    4.  
    5. int main(){
    6. QVector<int> vec(1000);
    7. vec[0]=0;
    8. for(int i=2; i<=2000000;i++){
    9. qDebug("Resizing to: %d", i*1000);
    10. try {
    11. vec.resize(1000*i);
    12. } catch(...){
    13. qDebug("Resize failed");
    14. exit(0);
    15. }
    16. qDebug("Success: %d", vec.size());
    17. }
    18.  
    19. return 0;
    20. }
    To copy to clipboard, switch view to plain text mode 

    I ran it and it crashed after resizing to 134218000 items which is about my physical memory capacity. The backtrace suggested checking out QVector<int>::realloc and so I did. It uses both new[] and realloc depending on the case. I don't know which of them causes a crash but looking at the code I'd point to realloc returning NULL.

    My suggestion is to report a suggestion to Trolltech to check returned values from functions like malloc or realloc. As I wrote in the previous post, the actual decision to implement or ignore the suggestion might depend on the behaviour of those functions on different platforms. But it's worth to try.

  12. #12
    Join Date
    Feb 2006
    Location
    Romania
    Posts
    2,744
    Thanks
    8
    Thanked 541 Times in 521 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    Allocating 16GB of data will ALWAYS fail on 32bit architecture, as you can't address more than 4GB of memory at once
    Not true actually. All modern CPUs (at least Intel's) provide a feature called "Physical Address Extension" which allows 32 bit systems to have up to 64 GB of memory or more, of course, with appropriate support in the operating system's kernel. I know that Linux supports it and some server versions of Windows 2000 and 2003.

    Although there isn't any support in the processor's instruction set for addressing more than 4GB of memory, the operating system can provide support for this too. See AWE on Windows and mmap on Linux.

  13. #13
    Join Date
    Oct 2007
    Posts
    6
    Qt products
    Qt4
    Platforms
    MacOS X Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    Additionally I tested QWidget class under same low memory conditions and it is also crashes.
    I’ll definitively submit bug report about all these problems to Trolltech Task Tracker.

    My current conclusion is that Qt library not handling memory allocation failure at all and I don't see any workaround for the problem.

  14. #14
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Memory allocation failure and crash in QVector

    Quote Originally Posted by marcel View Post
    Not true actually. All modern CPUs (at least Intel's) provide a feature called "Physical Address Extension" which allows 32 bit systems to have up to 64 GB of memory or more, of course, with appropriate support in the operating system's kernel. I know that Linux supports it and some server versions of Windows 2000 and 2003.
    But can you allocate more than 4GB as a single chunk? 32bit pointer is a 32bit pointer, it only supports 2^32 different addresses. You can have more than 4GB of physical memory, but does the same apply to a continuous area of virtual memory? I doubt that. I think even windowing extensions you mention will fail to do that.

  15. #15
    Join Date
    Feb 2006
    Location
    Romania
    Posts
    2,744
    Thanks
    8
    Thanked 541 Times in 521 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    It certainly is possible. Of course, I have no means to test this myself, but MSDN says the purpose of AWE is exactly to provide to a 32 bit process the means to access more than 4 GB.
    The translation from 32 bit to 36 bit memory address is done by the os kernel by using the PAE support in the processor.

  16. #16
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Memory allocation failure and crash in QVector

    Quote Originally Posted by marcel View Post
    MSDN says the purpose of AWE is exactly to provide to a 32 bit process the means to access more than 4 GB.
    That "W" in AWE means that you see only a part of the data at a time, so it's more like a memory mapping than allocation.

  17. #17
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,359
    Thanks
    3
    Thanked 5,015 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: Memory allocation failure and crash in QVector

    Quote Originally Posted by marcel View Post
    ...is exactly to provide to a 32 bit process the means to access more than 4 GB.
    Yes, but not more than 4GB at once.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.