Bug 852021

Summary: rsyslog.service cancels emergency.target making it impossible to enter emergency shell
Product: [openSUSE] openSUSE 13.1 Reporter: Andrei Borzenkov <arvidjaar>
Component: BasesystemAssignee: systemd maintainers <systemd-maintainers>
Status: RESOLVED DUPLICATE QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P5 - None CC: aeszter, crrodriguez, forgotten_o5N6HWuMhD, hpj, jjletho67-esus, werner
Version: Final   
Target Milestone: ---   
Hardware: Other   
OS: openSUSE 13.1   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: systemd debug log entering emeregency shell

Description Andrei Borzenkov 2013-11-24 17:41:37 UTC 
Created attachment 568839 [details]
systemd debug log entering emeregency shell

User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0

As was found in bnc#851993, if rsyslog service is active it may be impossible to enter emergency shell. The reason is, rsyslog.service requires basic.target which requires sysinit.target which conflicts with emergency.{target,service}. So if syslog.service is triggered by connection to syslog.socket it tries to start basic.target which cancels pending emergency shell.

Nov 23 23:20:05 server systemd[1]: Started Emergency Shell.
Nov 23 23:20:05 server systemd[1]: Starting Emergency Mode.
Nov 23 23:20:05 server systemd[1]: emergency.target changed dead -> active
Nov 23 23:20:05 server systemd[1]: Job emergency.target/start finished, result=done
Nov 23 23:20:05 server systemd[1]: Reached target Emergency Mode.
...
Nov 23 23:20:05 server systemd[1]: Incoming traffic on syslog.socket
Nov 23 23:20:05 server systemd[1]: Trying to enqueue job rsyslog.service/start/replace
Nov 23 23:20:05 server systemd[1]: Installed new job rsyslog.service/start as 541
Nov 23 23:20:05 server systemd[1]: Installed new job basic.target/start as 549
Nov 23 23:20:05 server systemd[1]: Installed new job sysinit.target/start as 550
...
Nov 23 23:20:05 server systemd[1]: Stopping Emergency Mode.
Nov 23 23:20:05 server systemd[1]: emergency.target changed active -> dead
Nov 23 23:20:05 server systemd[1]: Job emergency.target/stop finished, result=done
Nov 23 23:20:05 server systemd[1]: Stopped target Emergency Mode.
Nov 23 23:20:05 server systemd[1]: Stopping Emergency Shell...

Full log in attachment.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.
Comment 1 Cristian Rodríguez 2013-11-26 04:34:02 UTC 
not an rsyslog problem but systemd's . 

d420282b28f50720e233ccb1c02547c562195653 upstream.. needs new unit directive OnFailureJobMode..
Comment 3 Forgotten User o5N6HWuMhD 2013-11-26 17:51:37 UTC 
fyi @ IRC #systemd,

Q: What version of systemd will that (directive) show up in?  Is that determined yet? Distro is currently @ systemd-208-6.1.x86_64 ...

A:
Presumably, v209
...
It will be a while before v209 is out. There's lots of stuff to stabilize related to DBus work. Also, all the network work needs to settle. Lots of activity. :-)
...
It may be possible to backport the change to v208.
...
Backports are done by distros.
Comment 4 Andrei Borzenkov 2013-11-26 18:35:11 UTC 
@A R

Please test updated package. Assuming x86 platform.

zypper ar obs://home:arvidjaar:bnc:852021/standard bnc852021
zypper refresh bnc852021
zypper dup -r bnc852021

reboot and verify that system boots. Then enable back syslog.service (systemctl enable rsyslog.service), uncomment DEVICE lines in /etc/mdadm.conf and reboot. We should be back to your original configuration which hung at emergency mode; can you enter it now?

(In reply to comment #1)
> not an rsyslog problem 
I never said it was :)
Comment 5 Forgotten User o5N6HWuMhD 2013-11-26 18:38:10 UTC 
@Andrey 

I've only x86_64 to test ... 

Can you build/enable for that arch?
Comment 6 Andrei Borzenkov 2013-11-26 18:39:51 UTC 
(In reply to comment #5)
> I've only x86_64 to test ... 
> 
> Can you build/enable for that arch?

it is. x86 as opposed to arm or ppc. I did not say i386 :)
Comment 7 Forgotten User o5N6HWuMhD 2013-11-27 01:15:58 UTC 
> zypper ar obs://home:arvidjaar:bnc:852021/standard bnc852021
> zypper refresh bnc852021
> zypper dup -r bnc852021

zypper se -s | grep ^i | grep bnc852021
	i | libgudev-1_0-0                     | package    | 208-13.1        | x86_64 | bnc852021
	i | libudev-devel                      | package    | 208-13.1        | x86_64 | bnc852021
	i | libudev1                           | package    | 208-13.1        | x86_64 | bnc852021
	i | systemd                            | package    | 208-13.1        | x86_64 | bnc852021
	i | systemd-rpm-macros                 | package    | 2-13.1          | noarch | bnc852021
	i | systemd-sysvinit                   | package    | 208-13.1        | x86_64 | bnc852021
	i | udev                               | package    | 208-13.1        | x86_64 | bnc852021

> reboot and verify that system boots

it does

> Then enable back syslog.service (systemctl enable rsyslog.service)

done

> uncomment DEVICE lines in /etc/mdadm.conf

done

> reboot

done

> We should be back to your original configuration which hung at emergency mode;

yes, it hangs, and ...

> can you enter it now?

I'm now offered the emergency prompt, can login, traverse mounted/mountable fs, make edits, etc.
Comment 8 Marco M. 2013-11-28 17:40:30 UTC 
I had an almost identical issue and the patch has worked fine for me, now the emergency shell comes up correctly.

I encontered this bug while trying to solve those two others:

https://bugzilla.novell.com/show_bug.cgi?id=832501
https://bugzilla.novell.com/show_bug.cgi?id=811830


Thank you!
Comment 9 Forgotten User o5N6HWuMhD 2013-11-30 06:58:35 UTC 
fyi, related discussion thread @

http://lists.freedesktop.org/archives/systemd-devel/2013-November/014675.html
Comment 10 Andrei Borzenkov 2013-12-02 02:36:44 UTC 
*** Bug 852232 has been marked as a duplicate of this bug. ***
Comment 11 Andrei Borzenkov 2013-12-02 02:38:18 UTC 
@Marco, @A R: the patch was enough; I now updated my package with additional workaround and would appreciate additional testing.
Comment 12 Marco M. 2013-12-03 19:23:56 UTC 
(In reply to comment #11)
> @Marco, @A R: the patch was enough; I now updated my package with additional
> workaround and would appreciate additional testing.

Tested, it works! The emergency shell comes out correctly. 

Just to be clear, i tested only the ability to show a working emergency shell when it is expected  (in my case when a needed filesystem or partition is missing)
Comment 13 Marius Tomaschewski 2014-01-23 10:03:13 UTC 
see above (comment #1ff) -> systemd.
Comment 14 Dr. Werner Fink 2014-01-23 10:43:27 UTC 
Ahh ... indeed this explains why I was not able to reproduce this problem for bnc #832220

*** This bug has been marked as a duplicate of bug 832220 ***
Comment 15 Dr. Werner Fink 2014-03-03 15:30:43 UTC 
(In reply to comment #2)

This is part of the patch

  0001-core-replace-OnFailureIsolate-setting-by-a-more-gene.patch

OK then this should work with the latest systemd from factory.
Comment 16 Andrei Borzenkov 2014-03-04 14:59:33 UTC 
(In reply to comment #15)
> 
> OK then this should work with the latest systemd from factory.

This bug is filed against 13.1. Anyone is going to provide fix for it?
Comment 17 Dr. Werner Fink 2014-03-04 15:28:22 UTC 
(In reply to comment #16)

I've a systemd-208 around but I'm not sure if this should be submitted to 13.1 as there had been bad experiences with such updates (compare bnc#866612 or bnc#854884) ... ?!?
Comment 18 Swamp Workflow Management 2016-02-03 14:23:22 UTC 
openSUSE-RU-2016:0320-1: An update that has 146 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 737690,742774,750845,818044,838475,841544,849870,852015,852021,852232,853293,854884,856389,856392,856858,857204,858864,859072,859365,860574,860937,861316,861489,863217,864745,864904,865834,866732,866933,867128,867663,867664,867840,868019,868230,868439,868931,869142,869603,872929,873432,873444,874665,875502,876587,876694,877021,877674,878525,880438,880732,881125,881559,881942,882393,882714,883565,884271,884403,885232,885288,886211,886599,886852,888178,888215,888612,889297,889357,890977,892096,892162,892300,893797,895087,896664,897799,897801,897803,898233,898240,898432,900558,901481,902240,902901,903009,903963,904214,904517,904828,905550,906709,906900,907318,907393,908476,909358,910643,911347,912030,912334,913517,916420,918118,919095,920195,921831,921898,921920,926169,927250,927457,928265,931388,932284,933365,933512,933521,933533,934077,934901,937512,937900,938908,939571,940264,941576,944132,944799,945282,947212,948458,948555,948705,949574,949683,949739,950510,951265,951663,953241,954336,954781,955635,961576
CVE References: 
Sources used:
openSUSE 13.1 (src):    systemd-210-40.1, systemd-mini-210-40.1