Bug 852232

Summary: some non fatal failure during boot results in an endless loop due to defect emergency mode
Product: [openSUSE] openSUSE 13.1 Reporter: Hans-Peter Jansen <hpj>
Component: BasesystemAssignee: E-mail List <bnc-team-screening>
Status: RESOLVED DUPLICATE QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P5 - None CC: arvidjaar, ohering
Version: Final   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE 13.1   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: screenshot: error loop
screenshot: no login
screenshot: still no login
exit and reenter emergency mode

Description Hans-Peter Jansen 2013-11-25 21:39:44 UTC
Created attachment 569041 [details]
screenshot: error loop 

User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0

On a fresh installation, yast failed to create a fstab (#852027). 

While creating one myself, I also added some nfs mounts. For some reason, these mounts don't work correctly (due to a missing rpcbind, nfs service tries to access /etc/init.d/rpcbind, but that isn't available, hence it fails).

Now the whole system is installed in /, so emergency should work, but it spits out some errors in an endless loop. 

Scrrenshot attached.

Reproducible: Always

Steps to Reproduce:
1. add some mount target, that does not exist or results in some other failure
2. reboot
Actual Results:  
results in error loop


Expected Results:  
enter emergency mode
Comment 1 Andrei Borzenkov 2013-11-26 02:54:56 UTC
Probably duplicate of https://bugzilla.novell.com/show_bug.cgi?id=852021
Comment 2 Hans-Peter Jansen 2013-11-26 10:23:15 UTC
I thought about adding this as a comment, but decided, that the problem domain is better described from a UX POV, especially regarding the outcome...

I didn't dig deeper into the basic issues, I have to admit. Still approaching systemd, and still not satisfied from its behavior.
Comment 3 Andrei Borzenkov 2013-11-27 02:34:42 UTC
Please test suggested fix in https://bugzilla.novell.com/show_bug.cgi?id=852021
Comment 4 Hans-Peter Jansen 2013-11-27 14:55:44 UTC
I cannot trigger the failure again with these packages in place. Unfortunately, no amount of tinkering with fstab with trigger the emergency mode anymore. Even removing had no consequences. Will restore the original packages and redo the checks today evening.
Comment 5 Andrei Borzenkov 2013-11-27 15:49:34 UTC
(In reply to comment #4)
> I cannot trigger the failure again with these packages in place. Unfortunately,
> no amount of tinkering with fstab with trigger the emergency mode anymore. Even
> removing had no consequences. Will restore the original packages and redo the
> checks today evening.

mkdir /test
echo "/dev/no-such-device /test ext2 defaults 1 2" >> /etc/fstab
reboot

If this does not trigger emergency mode, you have some other problem :)
Comment 6 Hans-Peter Jansen 2013-11-28 17:00:08 UTC
Created attachment 569565 [details]
screenshot: no login

Okay, that did the trick (in terms of a hanging boot).

If I don't do anything, the system keeps showing the boot screen. I would expect a switch to the first console.

Hitting escape, it shows the emergency mode welcome message, but unfortunately, it doesn't ask for a password as usual. I tried to enter it anyway, but that doesn't helped either.

Esc toggles between the screens, but no login.

See screenshot.

Will fix with rescue system..
Comment 7 Andrei Borzenkov 2013-11-28 17:10:30 UTC
(In reply to comment #6)
> Hitting escape, it shows the emergency mode welcome message, but unfortunately,
> it doesn't ask for a password as usual.

I do not understand - is it with or without my patch?
Comment 8 Hans-Peter Jansen 2013-11-28 17:29:33 UTC
With, of course. Without it, it is already none to be defect..
Comment 9 Andrei Borzenkov 2013-11-28 17:37:16 UTC
Please show output of "rpm -qi systemd".
Comment 10 Hans-Peter Jansen 2013-11-28 19:59:45 UTC
Name        : systemd
Version     : 208
Release     : 14.1
Architecture: x86_64
Install Date: Mi 27 Nov 2013 11:45:03 CET
Group       : System/Base
Size        : 6239327
License     : LGPL-2.1+
Signature   : RSA/SHA1, Di 26 Nov 2013 19:49:02 CET, Key ID d784d3e91d5c5303
Source RPM  : systemd-208-14.1.src.rpm
Build Date  : Di 26 Nov 2013 19:47:47 CET
Build Host  : build35
Relocations : (not relocatable)
Vendor      : obs://build.opensuse.org/home:arvidjaar
URL         : http://www.freedesktop.org/wiki/Software/systemd
Summary     : A System and Session Manager
Description :
Systemd is a system and service manager, compatible with SysV and LSB
init scripts for Linux. systemd provides aggressive parallelization
capabilities, uses socket and D-Bus activation for starting services,
offers on-demand starting of daemons, keeps track of processes using
Linux cgroups, supports snapshotting and restoring of the system state,
maintains mount and automount points and implements an elaborate
transactional dependency-based service control logic. It can work as a
drop-in replacement for sysvinit.
Distribution: home:arvidjaar:bnc:852021
Comment 11 Andrei Borzenkov 2013-11-28 20:08:23 UTC
It was confirmed as working by two users, so you may be indeed facing some different problem. I guess, you are using plymouth? If yes, could you try with plymouth disabled (plymouth.enable=0 on kernel command line), whether you can reproduce this?
Comment 12 Hans-Peter Jansen 2013-11-28 20:42:31 UTC
Created attachment 569573 [details]
screenshot: still no login

Probably. Disabling plymouth does not change the behavior in question.
Comment 13 Andrei Borzenkov 2013-11-28 20:46:49 UTC
I noticed that you are using apparmor. Can you disable it to make sure it does not interfere?
Comment 14 Hans-Peter Jansen 2013-11-28 22:10:37 UTC
same behavior after "chkconfig boot.apparmor off"
Comment 15 Andrei Borzenkov 2013-11-30 11:06:14 UTC
Yes, I reproduced it. Upstream patch was not enough. Could you please test updated package (just run "zypper up", should pick it up). It contains additional workaround that fixed it for me.
Comment 16 Hans-Peter Jansen 2013-12-01 22:24:42 UTC
Created attachment 569754 [details]
exit and reenter emergency mode

Andrey,

sorry for the late response. Yes, that did the trick: after a mount timeout, the emergency shell (ES) is started, the root pw was requested, and I was able to enter it.

What's a little disturbing is: exiting with Ctrl-D usually resulted in a reboot. The new default seems to try to boot into default mode. Interestingly, I edited the fstab in the emergency session (commented out the defect entry),  hit Ctrl-D, system tried continue booting and - systemd has thrown me into the ES _again_. Another login -> Ctrl-D -> booting... -> ES. Looks like it keeps some state of ES, which leads to this, hence reboot is the only meaningful way out.. Therefor, the default behavior of continue booting is suboptimal at best.

Anyway, you're making progress, but sorry for being so disillusioning.

Thank you,
Pete
Comment 17 Andrei Borzenkov 2013-12-02 02:36:43 UTC
(In reply to comment #16)
> sorry for the late response. Yes, that did the trick: after a mount timeout,
> the emergency shell (ES) is started, the root pw was requested, and I was able
> to enter it.
> 

OK, so it is the same problem as in bnc#852021, so I mark is as duplicate.

> What's a little disturbing is: exiting with Ctrl-D usually resulted in a
> reboot. The new default seems to try to boot into default mode.

That's comes from upstream.

> I edited the fstab in the emergency session (commented out the defect entry), 
> hit Ctrl-D, system tried continue booting and - systemd has thrown me into the
> ES _again_. Another login -> Ctrl-D -> booting... -> ES. Looks like it keeps
> some state of ES, which leads to this, hence reboot is the only meaningful way
> out.. Therefor, the default behavior of continue booting is suboptimal at best.

Yes. After changing /etc/fstab you need to call "systemctl daemon-reload" to re-read it and forget old "bad" entry.

Feel free to open new bug report re. suboptimal behavior, but let's not pile everything in one bug.

*** This bug has been marked as a duplicate of bug 852021 ***
Comment 18 Dr. Werner Fink 2014-02-06 12:06:51 UTC
*** Bug 862450 has been marked as a duplicate of this bug. ***
Comment 19 Swamp Workflow Management 2016-02-03 14:23:37 UTC
openSUSE-RU-2016:0320-1: An update that has 146 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 737690,742774,750845,818044,838475,841544,849870,852015,852021,852232,853293,854884,856389,856392,856858,857204,858864,859072,859365,860574,860937,861316,861489,863217,864745,864904,865834,866732,866933,867128,867663,867664,867840,868019,868230,868439,868931,869142,869603,872929,873432,873444,874665,875502,876587,876694,877021,877674,878525,880438,880732,881125,881559,881942,882393,882714,883565,884271,884403,885232,885288,886211,886599,886852,888178,888215,888612,889297,889357,890977,892096,892162,892300,893797,895087,896664,897799,897801,897803,898233,898240,898432,900558,901481,902240,902901,903009,903963,904214,904517,904828,905550,906709,906900,907318,907393,908476,909358,910643,911347,912030,912334,913517,916420,918118,919095,920195,921831,921898,921920,926169,927250,927457,928265,931388,932284,933365,933512,933521,933533,934077,934901,937512,937900,938908,939571,940264,941576,944132,944799,945282,947212,948458,948555,948705,949574,949683,949739,950510,951265,951663,953241,954336,954781,955635,961576
CVE References: 
Sources used:
openSUSE 13.1 (src):    systemd-210-40.1, systemd-mini-210-40.1