QtService: Segfault after fork

[chenz]

Hello,

I have an odd problem with QtService (http://qt.gitorious.org/qt-solutions) on Linux (Debian stable):

When starting my daemon during boot (via init script), the process receives a segfault in QMutex::unlock. This seems to happen *after* the background process is forked, since the service process is actually running once booting is completed.

I can not reproduce this by (re-) starting the service once the system is up, it happens only during boot. Also the segfault does not happen on every boot, but only about 80% of the time, so it seems to be a race condition.

The crash also happens when I remove all of my service implementation except for minimal skeleton code, so I'm pretty sure this problem is not a side effect of my code.

Program terminated with signal 11, Segmentation fault.
#0  0xb7209c6c in QMutex::unlock (this=0x9a07f88) at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.cpp:370
370	               "A mutex must be unlocked in the same thread that locked it.");

Program terminated with signal 11, Segmentation fault.
#0  0xb7209c6c in QMutex::unlock (this=0x9a07f88) at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.cpp:370
370	               "A mutex must be unlocked in the same thread that locked it.");

To copy to clipboard, switch view to plain text mode 

Backtrace:

Thread 2 (Thread 936):
#0  0xb7800424 in __kernel_vsyscall ()
#1  0xb717ffcf in __pthread_cond_wait (cond=0x9a07f48, mutex=0x9a07f30) at pthread_cond_wait.c:153
#2  0xb72119fb in QWaitConditionPrivate::wait (this=0x9a07f30, time=4294967295)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qwaitcondition_unix.cpp:88
#3  0xb72117ff in QWaitCondition::wait (this=0x9a09b88, mutex=0x9a09b70, time=4294967295)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qwaitcondition_unix.cpp:160
#4  0xb7210fa3 in QThread::wait (this=0xb74b3794, time=4294967295)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qthread_unix.cpp:745
#5  0xb730349d in ~QProcessManager (this=0xb74b3794, __in_chrg=<value optimized out>)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/io/qprocess_unix.cpp:208
#6  0xb6f242df in __run_exit_handlers (status=0, listp=0xb7038304, run_list_atexit=true) at exit.c:78
#7  0xb6f2434f in *__GI_exit (status=0) at exit.c:100
#8  0xb6f0bcae in __libc_start_main (main=0x80520c4 <main>, argc=1, ubp_av=0xbfa8c674, init=0x8075480 <__libc_csu_init>, fini=0x8075470 <__libc_csu_fini>, 
    rtld_fini=0xb780f380 <_dl_fini>, stack_end=0xbfa8c66c) at libc-start.c:260
#9  0x08052031 in _start ()
 
Thread 1 (Thread 974):
#0  0xb7209c6c in QMutex::unlock (this=0x9a07f88) at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.cpp:370
#1  0xb71ed78a in QMutexLocker::unlock (this=0xb68b6250) at ../../include/QtCore/../../../qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.h:114
#2  0xb71ed747 in ~QMutexLocker (this=0xb68b6250, __in_chrg=<value optimized out>)
    at ../../include/QtCore/../../../qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.h:108
#3  0xb73669f1 in QEventDispatcherGlibPrivate (this=0x9a09fa8, context=0x0)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/kernel/qeventdispatcher_glib.cpp:299
#4  0xb7366da2 in QEventDispatcherGlib (this=0x9a09b10, parent=0x0)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/kernel/qeventdispatcher_glib.cpp:357
#5  0xb7210390 in QThreadPrivate::createEventDispatcher (data=0x9a09c20)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qthread_unix.cpp:272
#6  0xb72104d9 in QThreadPrivate::start (arg=0xb74b3794) at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qthread_unix.cpp:324
#7  0xb717b955 in start_thread (arg=0xb68b6b70) at pthread_create.c:300
#8  0xb6fc158e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Thread 2 (Thread 936):
#0  0xb7800424 in __kernel_vsyscall ()
#1  0xb717ffcf in __pthread_cond_wait (cond=0x9a07f48, mutex=0x9a07f30) at pthread_cond_wait.c:153
#2  0xb72119fb in QWaitConditionPrivate::wait (this=0x9a07f30, time=4294967295)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qwaitcondition_unix.cpp:88
#3  0xb72117ff in QWaitCondition::wait (this=0x9a09b88, mutex=0x9a09b70, time=4294967295)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qwaitcondition_unix.cpp:160
#4  0xb7210fa3 in QThread::wait (this=0xb74b3794, time=4294967295)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qthread_unix.cpp:745
#5  0xb730349d in ~QProcessManager (this=0xb74b3794, __in_chrg=<value optimized out>)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/io/qprocess_unix.cpp:208
#6  0xb6f242df in __run_exit_handlers (status=0, listp=0xb7038304, run_list_atexit=true) at exit.c:78
#7  0xb6f2434f in *__GI_exit (status=0) at exit.c:100
#8  0xb6f0bcae in __libc_start_main (main=0x80520c4 <main>, argc=1, ubp_av=0xbfa8c674, init=0x8075480 <__libc_csu_init>, fini=0x8075470 <__libc_csu_fini>, 
    rtld_fini=0xb780f380 <_dl_fini>, stack_end=0xbfa8c66c) at libc-start.c:260
#9  0x08052031 in _start ()

Thread 1 (Thread 974):
#0  0xb7209c6c in QMutex::unlock (this=0x9a07f88) at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.cpp:370
#1  0xb71ed78a in QMutexLocker::unlock (this=0xb68b6250) at ../../include/QtCore/../../../qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.h:114
#2  0xb71ed747 in ~QMutexLocker (this=0xb68b6250, __in_chrg=<value optimized out>)
    at ../../include/QtCore/../../../qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qmutex.h:108
#3  0xb73669f1 in QEventDispatcherGlibPrivate (this=0x9a09fa8, context=0x0)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/kernel/qeventdispatcher_glib.cpp:299
#4  0xb7366da2 in QEventDispatcherGlib (this=0x9a09b10, parent=0x0)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/kernel/qeventdispatcher_glib.cpp:357
#5  0xb7210390 in QThreadPrivate::createEventDispatcher (data=0x9a09c20)
    at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qthread_unix.cpp:272
#6  0xb72104d9 in QThreadPrivate::start (arg=0xb74b3794) at /home/build/dev/qt-everywhere-opensource-src-4.7.4/src/corelib/thread/qthread_unix.cpp:324
#7  0xb717b955 in start_thread (arg=0xb68b6b70) at pthread_create.c:300
#8  0xb6fc158e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

To copy to clipboard, switch view to plain text mode 

Does anyone have an idea what the problem might be?

Also, is there some kind of community support (bug tracker, discussion forum/list) for QtService/Qt-Solutions?

[wysota]

Can we see your code invoking the service?

[chenz]

I have now reduced this crash to a single line in the project file:

QT -= gui

QT -= gui

To copy to clipboard, switch view to plain text mode 

With this line, the crash occurs. Without this line, is does not... weird.

For some reason, qmake implicitly builds against QtGui, even though there is no obvious gui dependency in my code or qtservice. I really don't want my binary to depend on the gui libs, so i supress it using the line above...

Code:

#include <QCoreApplication>
 
#include "../qt-solutions/qtservice/src/qtservice.h"
 
class Service : public QtService<QCoreApplication>
{
public:
  Service(int argc, char** argv, const QString& name)
    : QtService<QCoreApplication>(argc, argv, name)
  {
  }
 
protected:
  void start()
  {
    logMessage("Start");
  }
 
  void stop()
  {
    logMessage("Stop");
  }
};
 
int main(int argc, char** argv)
{
  Service service(argc, argv, "Test service");
  return service.exec();
}

#include <QCoreApplication>

#include "../qt-solutions/qtservice/src/qtservice.h"

class Service : public QtService<QCoreApplication>
{
public:
  Service(int argc, char** argv, const QString& name)
    : QtService<QCoreApplication>(argc, argv, name)
  {
  }

protected:
  void start()
  {
    logMessage("Start");
  }

  void stop()
  {
    logMessage("Stop");
  }
};

int main(int argc, char** argv)
{
  Service service(argc, argv, "Test service");
  return service.exec();
}

To copy to clipboard, switch view to plain text mode 

Project:

SOURCES += main.cpp
# This line triggers crash:
QT -= gui
include(../qt-solutions/qtservice/src/qtservice.pri)

SOURCES += main.cpp
# This line triggers crash:
QT -= gui
include(../qt-solutions/qtservice/src/qtservice.pri)

To copy to clipboard, switch view to plain text mode 

Init script:

#! /bin/sh
 
### BEGIN INIT INFO
# Provides:          testservice
# Required-Start:    $network $local_fs $syslog dbus
# Required-Stop:     $network $local_fs $syslog dbus
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: testservice
# Description:       testservice
### END INIT INFO
 
EXE=/path/to/service
test -x $EXE || exit 0
 
USER=serviceuser
GROUP=serviceuser
 
. /lib/lsb/init-functions
 
case "$1" in
    start)
		log_daemon_msg "Starting test service" "testservice"
		if start-stop-daemon --start --quiet --oknodo --chuid ${USER}:${GROUP} --exec $EXE; then
			log_end_msg 0
		else
			log_end_msg 1
		fi
		;;
    stop)
		log_daemon_msg "Shutting down test service" "testservice"
		if start-stop-daemon --stop --quiet --oknodo --exec $EXE; then
			log_end_msg 0
		else
			log_end_msg 1
		fi
		;;
    restart)
		$0 stop
		$0 start
		;;
    *)
		log_action_msg "Usage: $0 {start|stop|restart}"
		exit 1
		;;
esac
 
exit 0

#! /bin/sh

### BEGIN INIT INFO
# Provides:          testservice
# Required-Start:    $network $local_fs $syslog dbus
# Required-Stop:     $network $local_fs $syslog dbus
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: testservice
# Description:       testservice
### END INIT INFO

EXE=/path/to/service
test -x $EXE || exit 0

USER=serviceuser
GROUP=serviceuser

. /lib/lsb/init-functions

case "$1" in
    start)
		log_daemon_msg "Starting test service" "testservice"
		if start-stop-daemon --start --quiet --oknodo --chuid ${USER}:${GROUP} --exec $EXE; then
			log_end_msg 0
		else
			log_end_msg 1
		fi
		;;
    stop)
		log_daemon_msg "Shutting down test service" "testservice"
		if start-stop-daemon --stop --quiet --oknodo --exec $EXE; then
			log_end_msg 0
		else
			log_end_msg 1
		fi
		;;
    restart)
		$0 stop
		$0 start
		;;
    *)
		log_action_msg "Usage: $0 {start|stop|restart}"
		exit 1
		;;
esac

exit 0

To copy to clipboard, switch view to plain text mode 

[wysota]

I don't see any reason for this code to crash. Are you using threads in your application (apart from the thread used internally by QtService)?

[chenz]

The code posted above is the complete test case that reproduces the crash, so no, I don't use any threads.

[wysota]

Do the examples bundled with QtService work properly?

[chenz]

This seems to be a static intitialization/race condition problem in Qt 4.7.4:

• Sometimes after I re-compile the problem goes away (this is also probably the cause why adding/removing QtGui has any effect)
• The problem goes away when I use Qt 4.7.4 compiled without glib
• It also goes away when I use Qt 4.8.0, which had some QThread race conditions fixed

[wysota]

Do the examples work?

[chenz]

The http server example shows the same behaviour.

[wysota]

In that case indeed the problem might be with Qt or Glib.

[martonmiklos]

The similar problem also affects me on Ubuntu 11.10 64 bit factory Qt 4.7.4 installed.

I would welcome any recommeddations about how should I create a Linux service using Qt with or without QtService.
Thanks!

[wysota]

With Linux it is easy since you don't need any extra treatment in the process itself to make it a daemon. The whole work is done by the init script. If you really want, you can just fork() the process to detach it from the terminal (see man daemon for more details).

[bob2oneil]

I have attempted to use the Linux API "daemon" to implement my application as a daemon. While the application runs to some extent, and can be viewed in the system manager, some of my custom threads do not seem to work.
These include an SNMP thread which makes a connection to the SNMP daemon, a TCP socket thread, etc. There are other threads which do seem to work This is for a 4.8 build dynamically linked.

Is there any particular issue with having a multithreaded Qt application work as a daemon, and specifically network related threads.

I was going to try the heavier weight QtService to see if it provides a solution.

In my application, I need to act as a daemon by default, but as a simple console application based on a command line argument.

Anyone have any thoughts or experience with either of these implementations?

[bob2oneil]

Some additional testing has revealed that none of my secondary threads run when my application is run as a daemon using the Linux "daemon" command (e.g. daemon(0, 0)), but execute without issue when the application is run as a console application. This application is a pure console application without any GUI, and uses only the Qt Core and Qt Networking libraries (version 4.8).

Is anyone aware of Qt secondary thread issues when using the Linux daemon command? Are there any workarounds?

It there any reason to believe that implementing a solution basd on QtService would fair any better?

[wysota]

Using daemon() has nothing to do with your threads. If your threads fail to work then it's likely because you're using some relative paths or something similar in your app.

[bob2oneil]

Thanks for you input. These secondary threads do not have any file system path content or issues that I am aware of. I have these secondary threads working now, and it required that the "daemon" call occur in main.cpp prior to the creation of the application object. Perhaps there is an explaination for this behavior.

[bob2oneil]

To strike a finer point on this issue, I have narrowed the problem down to a routine that I composed to detect currently running processes by process name to ensure that my application is run as a singleton.

This routine utilizes the QProcess() Qt class and invokes "pidof" to obtain a list of PIDs for a particular process name.

This appears to be problematic if invoked prior to the daemon() API call.

I am not sure why, but these are my observations under Qt 4.8 with dynamic linking.

[wysota]

What daemon() does is that it forks a new process (with a different process id) so at some point, until the parent process exits, you have two identical processes with different ids. Please read man fork for more details.

[bob2oneil]

Hi Wytold, master of all Qt knowledge.

I need my daemon to run at boot time without a user login. I have looked briefly at systemd as well as writing a boot script for inclusion into /etc/init.d. The target OS is likely to be a legacy version of Red Hat Enterprize Linux, and
it appears that systemd sort of began with Fedora 14. Do you have any experience with either of these or preferences.

I am also considering using the QtService class.

Trying to rationally evaluate the options:r development time and complexity versus benefits and legacy RHEL support.

I do not anticipate that my daemon would need to be started and restarted.

I also need my daemon to operate conditionally not like a daemon via a command line argument (i.e. as a console application). For this latter case, perhaps the startup mechanism would not be used.

[wysota]

You don't need systemd (whatever that is). You have sysvinit available. Just copy one of /etc/init.d/ scripts and modify it to fit your purpose. Or write your own, the Web is full of articles how to do that.