Results 1 to 9 of 9

Thread: Persistent queue?

  1. #1
    Join Date
    Jan 2006
    Posts
    368
    Thanks
    14
    Thanked 18 Times in 17 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Persistent queue?

    I am writing a small app, which basically needs to store messages and relay them to 3rd party servers. Basically, a QList where is push and pop data. However, my boss told me that the data must be also saved into disk, and memory only is not good enough.

    I can code something, maybe storing the whole QList into disk each time, or storing it into an SQLite3 and then removing stuff as needed. I am looking for something already existing.

    Please don't post messages about RabbitMQ or other high performance message queues. They are overkill for my needs (think: Using Oracle when you need an INI file).

    Thanks!

    I am writing a small app, which basically needs to store messages and relay them to 3rd party servers. Basically, a QList where is push and pop data. However, my boss told me that the data must be also saved into disk, and memory only is not good enough.

    I can code something, maybe storing the whole QList into disk each time, or storing it into an SQLite3 and then removing stuff as needed. I am looking for something already existing.

    Please don't post messages about RabbitMQ or other high performance message queues. They are overkill for my needs (think: Using Oracle when you need an INI file).

    Thanks!


    Added after 25 minutes:


    Replying to myself, as I am that cool:

    How about using http://stxxl.sourceforge.net/ ...? Its an STL implementation that saved the data into disk when needed.

    Might be an overkill... but this is the idea I am looking for, a simple API.
    Last edited by elcuco; 6th August 2015 at 14:57.

  2. #2
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Persistent queue?

    Simple serialization can also be done via QDataStream.
    Just needs QDataStream operators in the value classes and then you can stream a whole QList in and out of the stream.

    Cheers,
    _

  3. #3
    Join Date
    Jan 2008
    Location
    Alameda, CA, USA
    Posts
    5,230
    Thanks
    302
    Thanked 864 Times in 851 Posts
    Qt products
    Qt5
    Platforms
    Windows

    Default Re: Persistent queue?

    Replying to myself, as I am that cool
    I see people on the city streets talking to themselves all the time. I never realized how cool they were until I read this.

  4. #4
    Join Date
    Dec 2009
    Location
    New Orleans, Louisiana
    Posts
    791
    Thanks
    13
    Thanked 153 Times in 150 Posts
    Qt products
    Qt5
    Platforms
    MacOS X

    Default Re: Persistent queue?

    Quote Originally Posted by elcuco View Post
    Might be an overkill... but this is the idea I am looking for, a simple API.
    IMHO, you'll spend more time googling for an API or posting to this forum than it would take to serialize your QList to disk yourself.

    If you're using a database for other purposes, just add a table and insert/delete rows from the table and you queue/dequeue messages.

    If you're not already using a database, just write a file with the queued messages. You'll have to rewrite the file each time you queue a new message or dequeue a message. Perhaps look at QFileDevice::map for memory mapped files as well.

    Good luck.
    I write the best type of code possible, code that I want to write, not code that someone tells me to write!

  5. #5
    Join Date
    Mar 2009
    Location
    Brisbane, Australia
    Posts
    7,729
    Thanks
    13
    Thanked 1,610 Times in 1,537 Posts
    Qt products
    Qt4 Qt5
    Platforms
    Unix/X11 Windows
    Wiki edits
    17

    Default Re: Persistent queue?

    If you are going to write a backing file yourself, and it is important that messages do not get lost, then be careful how you do it. A temporary out of disk space condition can ruin your day if it breaks during write of a new queue file and you have overwritten the existing file.

  6. #6
    Join Date
    Dec 2009
    Location
    New Orleans, Louisiana
    Posts
    791
    Thanks
    13
    Thanked 153 Times in 150 Posts
    Qt products
    Qt5
    Platforms
    MacOS X

    Default Re: Persistent queue?

    Quote Originally Posted by ChrisW67 View Post
    If you are going to write a backing file yourself, and it is important that messages do not get lost, then be careful how you do it.
    Agreed, and thanks for pointing out. I should have been more specific, i.e. write new file, rename old file, rename new file, delete old file, etc.
    I write the best type of code possible, code that I want to write, not code that someone tells me to write!

  7. #7
    Join Date
    Dec 2009
    Location
    New Orleans, Louisiana
    Posts
    791
    Thanks
    13
    Thanked 153 Times in 150 Posts
    Qt products
    Qt5
    Platforms
    MacOS X

    Default Re: Persistent queue?

    Quote Originally Posted by jefftee View Post
    If you're not already using a database, just write a file with the queued messages. You'll have to rewrite the file each time you queue a new message or dequeue a message. Perhaps look at QFileDevice::map for memory mapped files as well.
    Sorry for replying to myself, I don't see that I can edit my prior post for some reason!

    After thinking about your question a bit, I would just add support for SQLITE to your app. It doesn't require an database server and is very easy to implement.
    I write the best type of code possible, code that I want to write, not code that someone tells me to write!

  8. #8
    Join Date
    Jan 2006
    Location
    Graz, Austria
    Posts
    8,416
    Thanks
    37
    Thanked 1,544 Times in 1,494 Posts
    Qt products
    Qt3 Qt4 Qt5
    Platforms
    Unix/X11 Windows

    Default Re: Persistent queue?

    Quote Originally Posted by jefftee View Post
    Agreed, and thanks for pointing out. I should have been more specific, i.e. write new file, rename old file, rename new file, delete old file, etc.
    http://doc.qt.io/qt-5/qsavefile.html

    Cheers,
    _

  9. The following user says thank you to anda_skoa for this useful post:

    jefftee (7th August 2015)

  10. #9
    Join Date
    Jan 2006
    Posts
    368
    Thanks
    14
    Thanked 18 Times in 17 Posts
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: Persistent queue?

    Thanks for the replies... and I did think of some of the things already mentioned here. Regarding disk full, I do have this as a basic assumption, near to "plug in the PC". Lets assume that if disk is full, I cannot do anything. Same as "new fails" - it's OK for the app to die, as I cannot do anything to remedy this.

    Another option I was thinking... is writing writing all the data received into file in fdisk, and them append a "u32" which means "how much did I read". By default is "0", meaning all is unread. To append, I
    Qt Code:
    1. fopen()
    To copy to clipboard, switch view to plain text mode 
    , and then
    Qt Code:
    1. fseek( filesize - sizeOf(u32))
    To copy to clipboard, switch view to plain text mode 
    , read the unread offset, and then fseek() again, append new data and write that offset again. When "consuming" data from the file, I just need to fseek() and then write only 8 bytes. IMHO this should be less IO then using SQlite. As this file is only managed by my app, I don't think this might break.

    What do you guys think, what can break with this setup?

    (2nd option is SQLite)

Similar Threads

  1. Persistent combobox entries.
    By spsingh in forum Qt Programming
    Replies: 2
    Last Post: 10th March 2010, 18:40
  2. QLineEdit persistent selection
    By elmo in forum Qt Programming
    Replies: 5
    Last Post: 5th September 2009, 14:15
  3. Persistent Editor Segmentation Fault
    By ArmanS in forum Qt Programming
    Replies: 0
    Last Post: 16th March 2009, 13:15
  4. Persistent widgets in QTableView.
    By kaushal_gaurav in forum Qt Programming
    Replies: 1
    Last Post: 1st August 2008, 21:06
  5. Help using the QTreeWidget persistent editor.
    By Billy Lee Black in forum Qt Programming
    Replies: 13
    Last Post: 18th July 2007, 20:59

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.