Advise sought on QEventLoop behavior

[TorAn]

I have a strange case where the pair of entry/exit calls works fine when breakpoint is set, but does not if there is no breakpoint. (all in debug mode).

I have QRunnable-derived class which also inherits from QEventLoop.
In the "run" method I call exec() and then, after some processing, the event is posted into this instance that calls "exit" in event() function.

Everything works if the breakpoint is set on "exit" call, I mean the event loop exits processing. However, without the breakpoint set, the eventloop never exits, even though the exit event is processed.

My event method is:

bool DataSourceProcess::event(QEvent* event)
{
	if (event->type() == csvEvent::type())
	{
		std::ofstream f ("c:\\test\\test.Text");
		f <<"exit";
		f.close();
		exit ();
	}
	return QObject::event(event);
}

bool DataSourceProcess::event(QEvent* event)
{
	if (event->type() == csvEvent::type())
	{
		std::ofstream f ("c:\\test\\test.Text");
		f <<"exit";
		f.close();
		exit ();
	}
	return QObject::event(event);
}

To copy to clipboard, switch view to plain text mode 

I tried to use "wakeUp()" call prior to "exit()" with no effect. With breakpoin set on exit - everything works. Without breakpoint the eventloop never exits.

I'll appreciate any advise or help in solving this problem (Qt 4.7.3, VS2010)

[Santosh Reddy]

How do you call run(), I mean do use thread pool or as a regular bock and wait call ?

[TorAn]

run method with details omitted for clarity.

void DataSourceProcess::run()
{
	res = addNewItems (); //addnewitem posts progress into this instance event loop, which in turn calls "exit()" as shown in the prevous post
	if (res)
	{
		emit onProgress("Entering event loop");
		exec();
		emit onProgress("Exited from event loop");
	}
 
	emit onProgress("cleanup") 
	if (_parent != NULL)
	{
		_sm->acquire(1);
		if (_sm->available() == 0)
			_parent->exit();
	}
}

void DataSourceProcess::run()
{
	res = addNewItems (); //addnewitem posts progress into this instance event loop, which in turn calls "exit()" as shown in the prevous post
	if (res)
	{
		emit onProgress("Entering event loop");
		exec();
		emit onProgress("Exited from event loop");
	}

	emit onProgress("cleanup") 
	if (_parent != NULL)
	{
		_sm->acquire(1);
		if (_sm->available() == 0)
			_parent->exit();
	}
}

To copy to clipboard, switch view to plain text mode 

Class declaration:

class DataSourceProcess : public QEventLoop, public QRunnable, public boost::noncopyable

class DataSourceProcess : public QEventLoop, public QRunnable, public boost::noncopyable

To copy to clipboard, switch view to plain text mode 

[wysota]

Ok, but how do you actually execute the runnable? And why do you inherit QEventLoop? To me it seems you'd be better with using QThread.

[TorAn]

void DataManager::launchDataProcess(const configDS& ds)
{
	DataSourceProcess* dsp = new DataSourceProcess(ds, _dmcenter);
	connect(dsp, SIGNAL(onProgress (QString)), this, SLOT(onProgressUpdate(QString)) );
	connect(dsp, SIGNAL(onComplete (DataSourceProcess*)), this, SLOT(onProgressComplete(DataSourceProcess*)) );
	_processes.push_back(dsp);
	QThreadPool::globalInstance()->start(dsp);
}

void DataManager::launchDataProcess(const configDS& ds)
{
	DataSourceProcess* dsp = new DataSourceProcess(ds, _dmcenter);
	connect(dsp, SIGNAL(onProgress (QString)), this, SLOT(onProgressUpdate(QString)) );
	connect(dsp, SIGNAL(onComplete (DataSourceProcess*)), this, SLOT(onProgressComplete(DataSourceProcess*)) );
	_processes.push_back(dsp);
	QThreadPool::globalInstance()->start(dsp);
}

To copy to clipboard, switch view to plain text mode 

I agree, QThread seems to be better, but this is an existing code. The issue shows up after new requirements were implemented on top of it. And I'd like to understand what's going on with QRunnable and QEventLoop.

---

Added after 1 19 minutes:

unsuccessfull test:

I've tried to isolate the issue with this simple use case. It implements console app with two methods of using QRunnable+QEventLoop: with events and with signal/slots.

Both exibit the same problem with posting events.

[wysota]

Does the runnable get executed at all? Did it occur to you that blocking a runnable with a semaphore on a single core machine means you're exhausting the thread pool and no other concurrent tasks can run until the one blocked wakes up and finishes its task? Not that the block with acquiring the semaphore makes ANY sense...

[TorAn]

Wysota,

First of all, thank you for reviewing the case.

Following are the answers, based not on the example code that I posted here, but on the actual code that lead me to ask my question.

1. Does the runnable get executed at all?
Yes, it does. I spend quite some time debugging the code and if it would not run I'd notice it well before wasting qtcenter.com user's time posting my question

Main() starts a thread with run() method as:

void myThread::run()
{
 
	while(QCoreApplication::startingUp ())
		sleep(1000);
 
 
	int inuse = 0;
	for (configOptions::container::const_iterator it = _options.datasources().begin(); it != _options.datasources().end(); ++it)
		if (it->isUsable())
			++inuse;
 
	if (inuse == 0)
		return;
 
 
	QSemaphore sm(inuse);
 
	for (configOptions::container::const_iterator it = _options.datasources().begin(); it != _options.datasources().end(); ++it)
		if (it->isUsable())
		{
			DataSourceProcess* dsp = new DataSourceProcess(*it, drh, this, &sm);
			QThreadPool::globalInstance()->start(dsp);
		}
 
	exec();
	qApp->exit(0);
}

void myThread::run()
{

	while(QCoreApplication::startingUp ())
		sleep(1000);


	int inuse = 0;
	for (configOptions::container::const_iterator it = _options.datasources().begin(); it != _options.datasources().end(); ++it)
		if (it->isUsable())
			++inuse;

	if (inuse == 0)
		return;


	QSemaphore sm(inuse);

	for (configOptions::container::const_iterator it = _options.datasources().begin(); it != _options.datasources().end(); ++it)
		if (it->isUsable())
		{
			DataSourceProcess* dsp = new DataSourceProcess(*it, drh, this, &sm);
			QThreadPool::globalInstance()->start(dsp);
		}

	exec();
	qApp->exit(0);
}

To copy to clipboard, switch view to plain text mode 

2. Did it occur to you that blocking a runnable with a semaphore on a single core machine means you're exhausting the thread pool and no other concurrent tasks can run until the one blocked wakes up and finishes its task?
No, it did not, at least not in my case. QRunnable instances in QThreadPool are not blocked when one of these instances calls QSemaphore->acquire (1)? In my case QSemaphore instance is created with the number of resources equal to the number of QRunnable instances and is used as a counter for exiting the parent thread's event loop when all QRunnable instances are done working.

QRunnable instance run method ends with:

{
...
	if (_parent != NULL) // parent thread
	{
	   _sm->acquire(1); // decrease number of available resources
                 if (_sm->available() == 0) // when nothing is available
	         _parent->exit();      // exit event loop of parent thread
	}
}

{
...
	if (_parent != NULL) // parent thread
	{
	   _sm->acquire(1); // decrease number of available resources
                 if (_sm->available() == 0) // when nothing is available
	         _parent->exit();      // exit event loop of parent thread
	}
}

To copy to clipboard, switch view to plain text mode 

3. Not that the block with acquiring the semaphore makes ANY sense...
I don't create any blockage in my case. Parent thread starts event loop and need to exit it when all QRunnable processes are done working. I'll appreciate it if you can suggest a better approach.

[wysota]

I don't create any blockage in my case. Parent thread starts event loop and need to exit it when all QRunnable processes are done working. I'll appreciate it if you can suggest a better approach.

void func(type1 param1, type2 param2, ...) {
  // implement your runnable code here or call some other class/method
}
void Thr::run() {
  QFutureSynchronizer<void> sync;
  for(int i=0;i<...;++i){
    sync.addFuture(QtConcurrent::run(func, param1, param2,...));
  }
  // QFutureSynchronizer waits for all threads to finish
}

void func(type1 param1, type2 param2, ...) {
  // implement your runnable code here or call some other class/method
}
void Thr::run() {
  QFutureSynchronizer<void> sync;
  for(int i=0;i<...;++i){
    sync.addFuture(QtConcurrent::run(func, param1, param2,...));
  }
  // QFutureSynchronizer waits for all threads to finish
}

To copy to clipboard, switch view to plain text mode 

"Thr" doesn't need an event loop. Actually the whole thread is not needed for anything at all, you can have the same logic without it, for instance:

Result func(const Data &data) {
  // implement your logic here
}
 
void MainThread::someFunc() {
 
  QList<Data> data;
  data << Data(...);
  data << Data(...);
  data << Data(...);
  data << Data(...);
  QFutureWatcher<Result> *watcher = new QFutureWatcher<Result>(this);
  connect(watcher, SIGNAL(finished()), SLOT(processingDone()));
  QFuture<Result> future = QtConcurrent::mapped(data, func);
  watcher->setFuture(future);
}

Result func(const Data &data) {
  // implement your logic here
}

void MainThread::someFunc() {

  QList<Data> data;
  data << Data(...);
  data << Data(...);
  data << Data(...);
  data << Data(...);
  QFutureWatcher<Result> *watcher = new QFutureWatcher<Result>(this);
  connect(watcher, SIGNAL(finished()), SLOT(processingDone()));
  QFuture<Result> future = QtConcurrent::mapped(data, func);
  watcher->setFuture(future);
}

To copy to clipboard, switch view to plain text mode 

As for your original code -- you can't call parent thread's exit() method from within a different thread. The semaphore thingy also makes no sense, a simple QAtomicInt would do. Better yet, you should exit the event loop from the actual thread when all work is done.

Furthermore from what I see you exit the application immediately after the work is done, so you might as well use blockingMapped() (or even blockingMapReduced() to write results to the file as they come) in my QtConcurrent code and forget about the future watcher.

[TorAn]

I have completed the use case sample with the QThread-derived class. It works as expected. It is even more interesting in respect to 2008 discussion here, where the responder is Thiago Macieira from Nokia. It essentially states that combination of QRunnable and QEventLoop provides the same functionality as QThread. I don't think that it is based on the sample code.

I'll be very interested to hear comments.

Wysota:

Your last code snipped is very elegant. I did not use these classes at all in my practice. Thank you very much for your review.

[Santosh Reddy]

How do you call run(), I mean do use thread pool or as a regular bock and wait call ?

You did not answer this. How and where do you call run()?

[wysota]

He did answer that, in post #5, line 7.

[TorAn]

Per Wysota's example of the QConcurrent-based solution I am now looking at QFuture usage. I don't understand how to implement following scenario using this framework.

My application processes various data files and submits it to the website for storage. Each data file comes from a separate provider. So, in respect to QConcurrent, I can describe operation for data processing using functor that I can pass to QtConcurrent::map. But this instance has to make multiple requests to the website via webservice, wait for the response and then decide what to do next - either stop or continue to send data until it finishes. If more data is available it makes another request to send data and so on.

So, within this function object I have to implement waiting mechanism. Is my scenario a good candidate for QConcurrent or I should stick with direct threads?

[wysota]

Why do you want threads there at all? You can do all that in a single thread.

[TorAn]

There is more to the use case that make threading a right choice. What I don't understand is if QtConcurrent provieds support beyond the sequential tasks.

[wysota]

There is more to the use case that make threading a right choice.

Name one thing, please.

What I don't understand is if QtConcurrent provieds support beyond the sequential tasks.

QtConcurrent doesn't care about the nature of tasks it executes.

[TorAn]

For example, data acquisition processes for individual datasources can take from 3 min to 2 hours, depending on the datasource. There around 10 datasources that have to be processed, starting at the same time.

I understand that QtConcurrent can process any task. But when the task that is processed is event or signal driven I have to organize it in the task instance and that brings back eventloop into the task.

[wysota]

For example, data acquisition processes for individual datasources can take from 3 min to 2 hours, depending on the datasource. There around 10 datasources that have to be processed, starting at the same time.

So why is it useful to do it in threads? All your threads will be doing most of the time is waiting for data to flow in. One thread is enough to handle all that without any effort.

I understand that QtConcurrent can process any task. But when the task that is processed is event or signal driven I have to organize it in the task instance and that brings back eventloop into the task.

Obviously, if you need events, you need an event loop. But you don't need an event loop for signals.

If I understand correctly what you are trying to do, then you are way overcomplicating things. The best strategy is to acquire the data from network (or whatever outside source you're using) in a single thread for all the sources and then process the data using QtConcurrent. This gives you the simplest and the most efficient code.

[TorAn]

Wysota,

I very much appreciate your review. I agree with the approach you suggested, part of it is actually implemented in the production code. Network communication is shared between threads, only data processing happends individually in a separate threads. All the reasons to keep it that way are relatively weak and I should redo the implementation using one thread for network and another for data processing.

Per the results of my use case sample I already made the transition from QRunnable/QEventLoop to QThread. It required a miniscule change and everything works now as expected. However, in the next round of development I will make the next transition, from separate QThreads to one QThread and QConcurrent maps.

Your advise and support is very helpful, thanks again.

[wysota]

However, in the next round of development I will make the next transition, from separate QThreads to one QThread and QConcurrent maps.

No, no QThread. The main thread will do just fine, you don't need an extra thread for networking.