Bug 787228

I could be interesting to boot with systemd.log_level=debug systemd.log_target=console to see what is going on.

I can't reproduce the issue, even after enabling VPN..

(In reply to comment #1)
> I could be interesting to boot with systemd.log_level=debug
> systemd.log_target=console to see what is going on.
> 
> I can't reproduce the issue, even after enabling VPN..

It looks like not bound to VPN. It spits out that there is a dependency problem with some service.

Should
kill -54 1
kill -59 1
do the job above at runtime? As it doesn't seem to have any effect?

hmm, according to documentation and if I still know how to count:
debug => kill -56 1 
console output => kill -61 1

(In reply to comment #3)
> hmm, according to documentation and if I still know how to count:
> debug => kill -56 1 

SIGRTMIN+22

/usr/include/asm/signal.h:
...
#define SIGRTMIN        32

32+22=54

> console output => kill -61 1

SIGRTMIN+27

27+32=59

Right?

(In reply to comment #4)
> Right?

Nope, got it now.

This happens also with poweroff. However I never recall to enable debug+console before kexec/poweroff. Anyway, what I see now is ntpd cannot be stopped for some reason (Stopping ntpd .. [FAILED]) and systemd says it is a poweroff dependency failure. Then network is shut down and it blocks. I'll try to remeber to enable debug+console next time.

Created attachment 520032 [details]
screen when stuck

This is how it looks. ntpd looks suspicious there...

(In reply to comment #7)
> This is how it looks. ntpd looks suspicious there...

It's not ntpd. When I disable it the issue still occurs. It happens only when network connection is activated. Then any of kexec, poweroff and reboot gets stuck.

Forgot to add that there is a plenty of messages when debug is enabled and nothing relevant in there. What should I be looking for?

could you try to do the procedure described in http://freedesktop.org/wiki/Software/systemd/Debugging#Diagnosing_Shutdown_Problems to get a full trace of what is going on ?

(In reply to comment #10)
> could you try to do the procedure described in
> http://freedesktop.org/wiki/Software/systemd/Debugging#Diagnosing_Shutdown_Problems
> to get a full trace of what is going on ?

Ok.

Regarding the first step in there, CTRL+ALT+DEL forces reboot to proceed even if it was stuck.

Created attachment 523233 [details]
debug log of hung reboot

(In reply to comment #10)
> to get a full trace of what is going on ?

Here you go.

in the attached trace, did you cancel the shutdown or anything like that ?

There is something suspiscious :

[49911.355122] systemd[1]: sys-devices-virtual-net-tun0.device changed plugged -> dead
[49911.362057] systemd[1]: Accepted connection on private bus.
[49911.362625] systemd[1]: Got D-Bus request: org.freedesktop.systemd1.Manager.RestartUnit() on /org/freedesktop/systemd1
[49911.362652] systemd[1]: Trying to enqueue job dnsmasq.service/restart/replace
[49911.362999] systemd[1]: Installed new job dnsmasq.service/restart as 1925
[49911.363007] systemd[1]: Job dbus.socket/stop finished, result=canceled
[49911.363016] systemd[1]: Installed new job dbus.socket/start as 1928
[49911.363022] systemd[1]: Job sysinit.target/stop finished, result=canceled
[49911.363031] systemd[1]: Installed new job sysinit.target/start as 1929
[49911.363037] systemd[1]: Job local-fs.target/stop finished, result=canceled
[49911.363043] systemd[1]: Installed new job local-fs.target/start as 1930
[49911.363048] systemd[1]: Job boot-efi.mount/stop finished, result=canceled
[49911.363054] systemd[1]: Installed new job boot-efi.mount/start as 1931
[49911.363060] systemd[1]: Installed new job fsck@dev-disk-by\x2did-ata\x2dINTEL_SSDSA2M080G2GC_CVPO0175040N080JGN\x2dpart1.service/start as 1932
[49911.363066] systemd[1]: Job umount.target/start finished, result=canceled
[49911.363074] systemd[1]: Job reboot.service/start finished, result=dependency
[49911.363390] systemd[1]: Job reboot.target/start finished, result=dependency
[49911.363399] systemd[1]: Job reboot.target/start failed with result 'dependency'.
[49911.363404] systemd[1]: Job reboot.service/start failed with result 'dependency'.

It looks like turning off the vpn is restarting dnsmasq (which was already off), which is restarting a number of services.

Could you try replacing "/etc/init.d/dnsmasq restart" by "/etc/init.d/dnsmasq try-restart" in  /etc/openvpn/client.down ?

(In reply to comment #13)
> in the attached trace, did you cancel the shutdown or anything like that ?

No, no, it is what it does w/o my intervention.

> It looks like turning off the vpn is restarting dnsmasq (which was already
> off), which is restarting a number of services.
> 
> Could you try replacing "/etc/init.d/dnsmasq restart" by "/etc/init.d/dnsmasq
> try-restart" in  /etc/openvpn/client.down ?

Yeah, that fixed it. Should we update this document:
https://wiki.innerweb.novell.com/index.php/Services_Team/Policies/openVPN/client_setup
?

(In reply to comment #14)
> (In reply to comment #13)

> > It looks like turning off the vpn is restarting dnsmasq (which was already
> > off), which is restarting a number of services.
> > 
> > Could you try replacing "/etc/init.d/dnsmasq restart" by "/etc/init.d/dnsmasq
> > try-restart" in  /etc/openvpn/client.down ?
> 
> Yeah, that fixed it. Should we update this document:
> https://wiki.innerweb.novell.com/index.php/Services_Team/Policies/openVPN/client_setup
> ?

Done.

closing as "fixed"

Neat, so I can reboot after a half year :). Thanks.

just for the record, upstream has just fixed this issue properly, by creating transactions which can't be cancelled automatically (only with a command), for stuff like reboot, shutdown : a service being started at shutdown would no longer stop reboot transaction..

*** Bug 812541 has been marked as a duplicate of this bug. ***

Just a quick question: will there be or has there been update for 12.3 including this fix? I have just had a situation where this bug prevented a reboot on ~30 machines :-(

(In reply to comment #19)
> Just a quick question: will there be or has there been update for 12.3
> including this fix? I have just had a situation where this bug prevented a
> reboot on ~30 machines :-(

No, it can't be backported. You have to find which service is being started at shutdown and prevent that.