Persistent queue?

[elcuco]

I am writing a small app, which basically needs to store messages and relay them to 3rd party servers. Basically, a QList where is push and pop data. However, my boss told me that the data must be also saved into disk, and memory only is not good enough.

I can code something, maybe storing the whole QList into disk each time, or storing it into an SQLite3 and then removing stuff as needed. I am looking for something already existing.

Please don't post messages about RabbitMQ or other high performance message queues. They are overkill for my needs (think: Using Oracle when you need an INI file).

Thanks!

I am writing a small app, which basically needs to store messages and relay them to 3rd party servers. Basically, a QList where is push and pop data. However, my boss told me that the data must be also saved into disk, and memory only is not good enough.

I can code something, maybe storing the whole QList into disk each time, or storing it into an SQLite3 and then removing stuff as needed. I am looking for something already existing.

Please don't post messages about RabbitMQ or other high performance message queues. They are overkill for my needs (think: Using Oracle when you need an INI file).

Thanks!

---

Added after 25 minutes:

Replying to myself, as I am that cool:

How about using http://stxxl.sourceforge.net/ ...? Its an STL implementation that saved the data into disk when needed.

Might be an overkill... but this is the idea I am looking for, a simple API.

[anda_skoa]

Simple serialization can also be done via QDataStream.
Just needs QDataStream operators in the value classes and then you can stream a whole QList in and out of the stream.

Cheers,
_

[d_stranz]

Replying to myself, as I am that cool

I see people on the city streets talking to themselves all the time. I never realized how cool they were until I read this.

[jefftee]

Might be an overkill... but this is the idea I am looking for, a simple API.

IMHO, you'll spend more time googling for an API or posting to this forum than it would take to serialize your QList to disk yourself.

If you're using a database for other purposes, just add a table and insert/delete rows from the table and you queue/dequeue messages.

If you're not already using a database, just write a file with the queued messages. You'll have to rewrite the file each time you queue a new message or dequeue a message. Perhaps look at QFileDevice::map for memory mapped files as well.

Good luck.

[ChrisW67]

If you are going to write a backing file yourself, and it is important that messages do not get lost, then be careful how you do it. A temporary out of disk space condition can ruin your day if it breaks during write of a new queue file and you have overwritten the existing file.

[jefftee]

If you are going to write a backing file yourself, and it is important that messages do not get lost, then be careful how you do it.

Agreed, and thanks for pointing out. I should have been more specific, i.e. write new file, rename old file, rename new file, delete old file, etc.

[jefftee]

If you're not already using a database, just write a file with the queued messages. You'll have to rewrite the file each time you queue a new message or dequeue a message. Perhaps look at QFileDevice::map for memory mapped files as well.

Sorry for replying to myself, I don't see that I can edit my prior post for some reason!

After thinking about your question a bit, I would just add support for SQLITE to your app. It doesn't require an database server and is very easy to implement.

[anda_skoa]

Agreed, and thanks for pointing out. I should have been more specific, i.e. write new file, rename old file, rename new file, delete old file, etc.

http://doc.qt.io/qt-5/qsavefile.html

Cheers,
_

[elcuco]

Thanks for the replies... and I did think of some of the things already mentioned here. Regarding disk full, I do have this as a basic assumption, near to "plug in the PC". Lets assume that if disk is full, I cannot do anything. Same as "new fails" - it's OK for the app to die, as I cannot do anything to remedy this.

Another option I was thinking... is writing writing all the data received into file in fdisk, and them append a "u32" which means "how much did I read". By default is "0", meaning all is unread. To append, I

fopen()

fopen()

To copy to clipboard, switch view to plain text mode 

, and then

fseek( filesize - sizeOf(u32))

fseek( filesize - sizeOf(u32))

To copy to clipboard, switch view to plain text mode 

, read the unread offset, and then fseek() again, append new data and write that offset again. When "consuming" data from the file, I just need to fseek() and then write only 8 bytes. IMHO this should be less IO then using SQlite. As this file is only managed by my app, I don't think this might break.

What do you guys think, what can break with this setup?

(2nd option is SQLite)