Results 1 to 10 of 10

Thread: QSemaphore: 1 Producer, 3 Consumers

  1. #1
    Join Date
    Mar 2007
    Location
    Bielefeld, Germany
    Posts
    9
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default QSemaphore: 1 Producer, 3 Consumers

    Hi,

    the basic problem I was facing is described here

    As my work progressed, I faced the need to link 1 producer thread to 3 consumer-threads derived from the same class.

    In my Widget Class, the semaphore and buffer are declared as followed (based on the Semaphore example and a single consumer), which worked:

    Qt Code:
    1. QStringList threadBuffer[300];
    2.  
    3. QSemaphore *freeListSlots(300);
    4. QSemaphore *usedListSlots;
    5.  
    6. Importer *importerThread;
    7. Converter *converterThread1;
    8. Converter *converterThread2;
    9. Converter *converterThread3;
    To copy to clipboard, switch view to plain text mode 

    My idea was, to let thread1 work on buffer[0] to buffer[99], thread2 on buffer[100] to buffer[199] and so on. The thread, that has no accessible data, should block until new data arrives.

    My "emergency plan" is, to use a different semaphore for each thread and provide a pointer to it in the consumer-constructor, then acquire and release based on the actual buffer-position. But if there was a nicer way, I'd prefer the alternative

    Greets,
    Daedalus

  2. #2
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,365
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    If each thread works on a different buffer, what do you need semaphores for? Either make all threads work on the whole buffer or use a wait condition.

  3. #3
    Join Date
    Mar 2007
    Location
    Bielefeld, Germany
    Posts
    9
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    The worker constantly fills the (circular) buffer. What is not clear to me from the documentation is, how to work with a number of reading threads on this buffer without running into undefined parts of it or reading data twice.

    For one consumer thread, my code was:

    Qt Code:
    1. Converter::run()
    2. {
    3. int converted = 0;
    4.  
    5. while(!abort){
    6. parentWindow->usedListSlots->acquire();
    7. processEntry(threadBuffer[converted]);
    8. parentWindow->freeListSlots->release();
    9. converted++;
    10. if (converted==300){
    11. converted=0;
    12. }
    13. }
    14. }
    To copy to clipboard, switch view to plain text mode 
    The semaphore has a max value of 300, one for each item.

    if I run 3 of these threads, I get random crashes, while the import/convert-loops are processed. So I thought, I'd have to work on different parts of the same buffer simultaneously.

    But maybe I just got something completely wrong from the example.
    Last edited by Daedalus; 31st March 2007 at 08:50.

  4. #4
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    Quote Originally Posted by Daedalus View Post
    The semaphore has a max value of 300, one for each item.
    Which means that all three threads can acquire it simultaneously, which is not safe, since the docs doesn't say that QStringList is thread-safe. Only one thread should be accessing the buffer at a time (unless you'll have a buffer that can be read simultaneously by several threads) and for this you need a mutex.

    The Converter::run() should look more or less like this:
    Qt Code:
    1. Converter::run()
    2. {
    3. while( ! abort ) {
    4. QString entry( buffer->get() );
    5. if( ! abort ) {
    6. processEntry( entry );
    7. }
    8. }
    9. }
    To copy to clipboard, switch view to plain text mode 

    Now you have to implement a Buffer class with get() and put() methods which will use proper synchronisation mechanisms (for example a mutex and wait conditions or mutex and semaphores).

    Such approach doesn't depend on number of threads that operate on a buffer. You can have several consumers and several producers at the same time.

  5. #5
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,365
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    Furthermore if you use QReadWriteLock with a proper buffer implementation (for example a circular buffer with separate indexes for reading and writing), you'll be able to consume and produce at the same time. Here you could use a semaphore to make sure you avoid buffer underruns.

  6. The following user says thank you to wysota for this useful post:

    Daedalus (1st April 2007)

  7. #6
    Join Date
    Mar 2007
    Location
    Bielefeld, Germany
    Posts
    9
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    Thank you for your explanation . In my opinion, the documentation is a bit short regarding this point.

  8. #7
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,365
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    Just one thing - Jacek and I talked about this yesterday and he said using a read write lock won't help in this situation as a writer would block all readers. Of course he's right.

  9. #8
    Join Date
    Mar 2007
    Location
    Bielefeld, Germany
    Posts
    9
    Thanks
    1
    Qt products
    Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    I think, as the writer just performs a RegEx operation and some checks, it wouldn't slow down processing too much, as the reader needs a lot of time creating AbstractModelItems from the gathered data (about 38000 from a ( rather small ) 2.000.000 line file, in a productive environment it may be 8-10 times that many). But to make sure, that every thread finished working on the buffer, before unblocking the writer would create longer delays I think, as some threads may just idle away, waiting for all other readers to finish.

  10. #9
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    5,372
    Thanks
    28
    Thanked 976 Times in 912 Posts
    Qt products
    Qt3 Qt4
    Platforms
    Unix/X11 Windows

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    Quote Originally Posted by Daedalus View Post
    In my opinion, the documentation is a bit short regarding this point.
    I think it's because there is no point in duplicating general knowledge in the docs. If you read any handbook for academic course on operating systems, you'll know how to use all of the synchronisation mechanisms, regardless which library implements them.

    Quote Originally Posted by Daedalus View Post
    to make sure, that every thread finished working on the buffer, before unblocking the writer would create longer delays I think, as some threads may just idle away, waiting for all other readers to finish.
    If each reader thread will block access to the buffer until it finishes processing its entry, you'll end up with one working thread and two idle ones all of the time. The key to the success in parallel processing is to limit the synchronisation to necessary minimum, because inside the synchronisation point you can have only sequential processing.

    Note that all the reader thread needs from the buffer is just a single string, so you have to block access to the buffer only for time needed to read that string and until the thread finishes processing that string it doesn't have to access the buffer, so in that time other threads can read from the buffer.

  11. #10
    Join Date
    Jan 2006
    Location
    Warsaw, Poland
    Posts
    33,365
    Thanks
    3
    Thanked 5,014 Times in 4,792 Posts
    Qt products
    Qt3 Qt4 Qt5 Qt/Embedded
    Platforms
    Unix/X11 Windows Android Maemo/MeeGo
    Wiki edits
    10

    Default Re: QSemaphore: 1 Producer, 3 Consumers

    Remember that it's better to protect data than to protect code, so synchronise only when you really access the shared data and release the lock right away after you're done with it. You can easily allow simoultaneous reading and writing to the buffer, the only problem is to synchronise all readers and all writers.

Similar Threads

  1. Producer Consumer...
    By Shuchi Agrawal in forum Newbie
    Replies: 1
    Last Post: 16th February 2007, 09:45
  2. Producer Consumer
    By ^NyAw^ in forum Qt Programming
    Replies: 16
    Last Post: 17th November 2006, 19:53

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Digia, Qt and their respective logos are trademarks of Digia Plc in Finland and/or other countries worldwide.