Bug 727771

Summary: systemd: shuts down network for no reason
Product: [openSUSE] openSUSE 12.1 Reporter: Jiri Slaby <jslaby>
Component: BasesystemAssignee: Frederic Crozat <fcrozat>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P5 - None CC: bugproxy, crrodriguez, d.a.van.delft, doerges, engineering, fcrozat, forgotten_hJWYsjvCQn, forgotten_ZZuOragHZ_, hannsj_uhl, hharun, james.knott, jdsn, kuehne.stefan, lnussel, mfilka, mt, per, ro, werner
Version: Factory   
Target Milestone: ---   
Hardware: Other   
OS: Other   
See Also: https://bugzilla.linux.ibm.com/show_bug.cgi?id=127836
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 696902    
Attachments: dmesg output with systemd debug turned on
Excerpt from /var/log/messages of eth0, four boots, last showing the ifdown
logging output obtained as described in comment 39
testconf

Description Jiri Slaby 2011-11-02 12:58:32 UTC
When booting with systemd (sysvinit is OK), I have no network.

With sysvinit:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
    inet6 fe80::5054:ff:fe12:3456/64 scope link 
       valid_lft forever preferred_lft forever

With systemd:
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff


If I try to start it, I see:
# systemctl start network.service 
Setting up (localfs) network interfaces:
    lo        
    lo        IP address: 127.0.0.1/8                                done
    eth0      device: Intel Corporation 82540EM Gigabit Ethernet Co
[  129.640681] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
    eth0      IP address: 10.0.2.15/24                               waiting
Waiting for mandatory devices:  eth0 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
8 
    eth0      device: Intel Corporation 82540EM Gigabit Ethernet Co
    eth0      IP address: 10.0.2.15/24
    eth0      is up                                                  done
Setting up service (localfs) network  .  .  .  .  .  .  .  .  .  .   done
# Shutting down (localfs) network interfaces:
    eth0      device: Intel Corporation 82540EM Gigabit Ethernet Co  done


Why it shuts it down almost immediatelly?


messages say:
ifup:     lo        
ifup:     lo        
ifup: IP address: 127.0.0.1/8  
ifup:  
ifup:     eth0      device: Intel Corporation 82540EM Gigabit Ethernet Co
kernel: [  129.640681] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
ifup:     eth0      
ifup: IP address: 10.0.2.15/24  
ifup:  
avahi-daemon[781]: Joining mDNS multicast group on interface eth0.IPv4 with address 10.0.2.15.
avahi-daemon[781]: New relevant interface eth0.IPv4 for mDNS.
avahi-daemon[781]: Registering new address record for 10.0.2.15 on eth0.IPv4.
avahi-daemon[781]: Registering new address record for fe80::5054:ff:fe12:3456 on eth0.*.
ifdown:     eth0      device: Intel Corporation 82540EM Gigabit Ethernet Co
avahi-daemon[781]: Withdrawing address record for 10.0.2.15 on eth0.
avahi-daemon[781]: Leaving mDNS multicast group on interface eth0.IPv4 with address 10.0.2.15.
avahi-daemon[781]: Interface eth0.IPv4 no longer relevant for mDNS.
avahi-daemon[781]: Withdrawing address record for fe80::5054:ff:fe12:3456 on eth
Comment 1 Frederic Crozat 2011-11-02 13:23:54 UTC
could you post dmesg output after booting with systemd.log_level=debug systemd.log_target=kmsg ?
Comment 2 Jiri Slaby 2011-11-02 13:26:33 UTC
Created attachment 459928 [details]
dmesg output with systemd debug turned on

(In reply to comment #1)
> could you post dmesg output after booting with systemd.log_level=debug
> systemd.log_target=kmsg ?

Sure.
Comment 3 Frederic Crozat 2011-11-02 13:46:30 UTC
systemctl show network.service | grep RemainAfterExit 

?
Comment 4 Jiri Slaby 2011-11-02 13:47:37 UTC
(In reply to comment #3)
> systemctl show network.service | grep RemainAfterExit 

RemainAfterExit=no
Comment 5 Danny van Delft 2011-11-03 11:30:27 UTC
Created attachment 460178 [details]
Excerpt from /var/log/messages of eth0, four boots, last showing the ifdown

Same results, but not always. With one VM under VMWare ESX, about one in four times at bootup the eth0 interface gets downed, after it was initially up. Another, more or less identical, VM under VMServer works fine up till now.

Excerpt from /var/log/messages attached
Comment 6 Frederic Crozat 2011-11-04 13:22:12 UTC
this bug is caused by fix for bnc#721426, which sets, in this case, network service as a "RemainAfterExit=no" service.

you have a static network configuration and when all processes are gone, service is seen as "dead" instead of "running".

So, we need to add one or two LSB extra headers to tell systemd this service should stay as "RemainAfterExit".

I propose adding 

X-Remain-After-Exit: yes/no

X-Pidfile: <path_to_pid_file_when_present>

When X-Pidfile: is present, it would imply RemainAfterExit=no (this is currently what is done when systemd detects "redhat" initscript header, where pidfile value can be specified).

When X-Pidfile isn't specified and X-Remain-After-Exit isn't specified, previous heuristic will remain for RemainAfterExit : if a process is running after services is started, RemainAfterExit will be "false", otherwise, it will be "true".

Opinions welcome.
Comment 7 Danny van Delft 2011-11-04 13:38:19 UTC
On my system at the moment RemainAfterExit=yes, and even then it apparently gets downed once in a while during bootup (see previous #5). Or is the "yes" the result of the manual ifup done after the bootup?
I have BTW no idea how RemainAfterExit gets set. AFAIK I haven't done anything to
get it set to "yes", as in #4 it is set on "no".
Comment 8 Ludwig Nussel 2011-11-04 13:49:16 UTC
(In reply to comment #6)
> So, we need to add one or two LSB extra headers to tell systemd this service
> should stay as "RemainAfterExit".
> 
> I propose adding 
> 
> X-Remain-After-Exit: yes/no
> 
> X-Pidfile: <path_to_pid_file_when_present>

Sounds good. Although it should be X-Systemd or X-SUSE I guess.
Comment 9 Frederic Crozat 2011-11-04 13:53:09 UTC
Better to use X-Systemd- (who knows, we could get it merged upstream, even if I wouldn't bet my life on it ;)

Danny> for LSB initscripts, RemainAfterExit is set according to the following heuristic (needed by bug bnc#721426):
- once lsb initscript has been started, systemd checks if there is still a process running in the associated cgroup : if there is, RemainAfterExit=no, otherwise, RemainAfterExit=yes. Usually, you can detect daemon this way. Unfortunately, for some scripts (like network script), you might still have process running for some time after startup, which will quit a little later, so RemainAfterExit is set to "no" and systemd thinks the "daemon" (which wasn't one) has terminated.
Comment 10 Frederic Crozat 2011-11-04 16:48:33 UTC
I've started implementing parsing of the following headers :
PIDFile (this one is already used in nscd package and it will be consistent with fedora header)
X-Systemd-RemainAfterExit
Comment 11 Frederic Crozat 2011-12-05 15:37:18 UTC
I've pushed support for PIDFile and X-Systemd-RemainAfterExit in home:fcrozat:systemd / systemd

please test (and add  X-Systemd-RemainAfterExit: true in /etc/init.d/network header)
Comment 12 Marius Tomaschewski 2011-12-05 17:14:57 UTC
I've added it to network and network-remotefs in Base:System/sysconfig (0.76).
Comment 13 Bernhard Wiedemann 2011-12-05 18:00:21 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/95483 Factory / sysconfig
Comment 14 Jiri Slaby 2011-12-05 20:58:33 UTC
(In reply to comment #11)
> I've pushed support for PIDFile and X-Systemd-RemainAfterExit in
> home:fcrozat:systemd / systemd
> 
> please test (and add  X-Systemd-RemainAfterExit: true in /etc/init.d/network
> header)

Looks OK. But is this normal?
 30635ms bootsplash-quit.service
  4999ms network.service
  1158ms apparmor.service

There is no splash here...
Comment 15 Frederic Crozat 2011-12-06 09:14:24 UTC
ohh, indeed, bootsplash-quit.service is always started, even if bootsplash-startup.service.

could you paste systemctl status bootsplash-startup.service output, just to be sure ?
Comment 16 Frederic Crozat 2011-12-07 10:42:29 UTC
could you test latest version of systemd from home:fcrozat:systemd/systemd ?

I've ensured bootsplash-startup/quit aren't started if both vga= and splash=silent aren't present on kernel commandline
Comment 17 J. Daniel Schmidt 2011-12-07 15:02:24 UTC
*** Bug 732123 has been marked as a duplicate of this bug. ***
Comment 18 Frederic Crozat 2011-12-09 14:38:35 UTC
sr 96122 pushed to openSUSE:12.1:Update:Test

requesting maintenance update for 12.1

we also need a maintenance update for sysconfig, once systemd has been pushed
Comment 19 Bernhard Wiedemann 2011-12-09 15:00:30 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/96122 12.1 / systemd
https://build.opensuse.org/request/show/96125 Factory / systemd
Comment 20 Bernhard Wiedemann 2011-12-09 18:00:29 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/96193 Factory / systemd
Comment 21 Bernhard Wiedemann 2011-12-12 17:00:36 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/96377 12.1 / systemd
Comment 23 Till Dörges 2011-12-15 19:23:58 UTC
I guess I stumbled across this issue, too, but I'm not entirely sure:

 - Static IP config. Everything working fine w/ systemd to begin with.
 - Then the apache2 service was enabled.
 - Upon reboot network would start briefly but shut down immediately.

The "fix" was to disable the Apache service.

W/ Apache enabled (and network not working) I get RemainAfterExit=no. W/o Apache (and network working) I get RemainAfterExit=yes.

If this is another symptom of this bug, why does Apache cause the network to shut down immediately?
Comment 24 Cristian Rodríguez 2011-12-17 14:43:38 UTC
Already fixed in SR 96377 and 96377
Comment 25 Frederic Crozat 2012-01-04 09:50:09 UTC
12.1 maintenance update has been released for systemd

Now, we need a maintenance update for sysconfig (reopening bug report until everything is fixed in 12.1)

Marius, could you handle it ? I see on OBS some other sr (#97040, #97208) are pending regarding sysconfig. Maybe we should integrate the systemd fix in the same update ?
Comment 26 Marius Tomaschewski 2012-01-04 11:10:48 UTC
(In reply to comment #25)
> 12.1 maintenance update has been released for systemd
> 
> Now, we need a maintenance update for sysconfig (reopening bug report until
> everything is fixed in 12.1)
> 
> Marius, could you handle it ? I see on OBS some other sr (#97040, #97208) are
> pending regarding sysconfig. Maybe we should integrate the systemd fix in the
> same update ?

I'd not disturb the running updates.

Further, I'm still testing some issues -- going to branch sysconfig and I'll
apply all patches to home:mtomaschewski:branches:openSUSE:12.1:Update:Test.
Comment 27 Marius Tomaschewski 2012-01-04 14:57:34 UTC
Test packages with fixes is available at:

http://download.opensuse.org/repositories/home:/mtomaschewski:/branches:/openSUSE:/12.1:/Update:/Test/standard/x86_64/
Comment 28 Marius Tomaschewski 2012-01-04 18:47:57 UTC
*** Bug 737641 has been marked as a duplicate of this bug. ***
Comment 29 Bernhard Wiedemann 2012-01-10 10:00:30 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/99577 12.1 / sysconfig
Comment 30 Bernhard Wiedemann 2012-01-10 11:00:15 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/99585 12.1 / sysconfig
Comment 31 Marius Tomaschewski 2012-01-10 12:55:54 UTC
sysconfig package submitted in:
request 99595 -> 11.3: bnc#697929,bnc#739338 [SWAMPID 44824]
request 99593 -> 11.4: bnc#697929,bnc#739338 [SWAMPID 44824]
request 99585 -> 12.1: bnc#739338,bnc#697929,bnc#733118,bnc#727771,bnc#734723
Comment 32 Marius Tomaschewski 2012-01-10 13:42:18 UTC
*** Bug 736059 has been marked as a duplicate of this bug. ***
Comment 33 Forgotten User hJWYsjvCQn 2012-01-11 17:40:16 UTC
After updating to 3.2 kernel i have the same problem (with & without your new sysconfig).
Turning back to sysvinit-init is not a solution, so can i help with debugging?
Comment 34 Forgotten User ZZuOragHZ_ 2012-01-11 21:36:11 UTC
Hi

I have the same problem, but the fix has been revoked, So I'm unable to test.

For me it works if I add:
#/etc/init.d/network start -o debug

DHCP works,
Fixed IP starts and immediately stops.

systemd seems way too complicated

Thanks
Comment 35 Marius Tomaschewski 2012-01-12 10:33:23 UTC
(In reply to comment #34)
> Hi
> 
> I have the same problem, but the fix has been revoked, So I'm unable to test.

Which fix has been revoked?

> For me it works if I add:
> #/etc/init.d/network start -o debug

This works because there is no systemd involved / no redirection done by
the /etc/rc.status script because of this guard:

   if test $PPID -ne 1 -a $# -eq 1 ; then

[when not started by systemd and only one arg given -> redirect to systemd]

but it causes another problem: systemd does not know anything about your
start and continue to consider it dead/failed/not yet started, even a
"rcnetwork status -o" will show it started.

The problem (part of) is IMO, that systemd a) tries to resolve dependencies
[generally a good thing], b) does not use status reported by the LSB scripts
(does not call /etc/init.d/network status), but own meaning instead.

Further, there is IMO no way to tell let it refresh the systemd state, e.g.
via "systemctl reset-failed". It is seems to not work for LSB scripts, e.g.
by executing the LSB status action...


Additional problems may be caused by hooks installed by samba-client or ntp,
that cause a systemd deadlock by executing init scripts that depend from the
network script via hooks inside of the network scripts (if-up.d, ...), see
bug 732930 alias bug 732395 alias bug 725503 alias ...

Workarounds are (until release of fixes for samba-client, ...):

 - move the /etc/sysconfig/network/if-*.d/21* links away
 - remove ntp-runtime from NETCONFIG_MODULES_ORDER
   in /etc/sysconfig/network/config.
Comment 36 Marius Tomaschewski 2012-01-12 10:52:14 UTC
BTW: I've noticed also, that it seems to bee required to execute:
        systemctl --system daemon-reload
after changes e.g. of BOOTPROTO=dhcp to BOOTPROTO=static. Sometimes
a boot with active resume=/dev/<swap-device> still shows some wrong
behavior and I have to reboot once and set "noresume" boot option
instead.
Comment 37 Marius Tomaschewski 2012-04-04 14:38:26 UTC
*** Bug 732696 has been marked as a duplicate of this bug. ***
Comment 38 Danny van Delft 2012-04-04 15:36:53 UTC
Well, seems not to be fixed for me: network still gets downed at boot. Have done zypper up to current, containing:

roxeli:/var/log # rpm -q sysconfig systemd
sysconfig-0.75.4-2.7.1.x86_64
systemd-37-3.11.1.x86_64

roxeli:/var/log # rpm -q --changelog systemd | head -1
* wo mrt 14 2012 fcrozat@suse.com

roxeli:/var/log # rpm -q --changelog sysconfig | head -1
* wo feb 29 2012 mt@suse.com

Excerpt from /var/log/messages, showing the ifdown:

Apr  4 17:14:00 roxeli network[632]: 27
Apr  4 17:14:00 roxeli network[632]: eth0      device: Intel Corporation 82545EM Gigabit Ethernet Co
Apr  4 17:14:00 roxeli network[632]: eth0      IP address: 192.168.1.77/24
Apr  4 17:14:00 roxeli network[632]: eth0      is up
Apr  4 17:14:00 roxeli network[632]: ..doneSetting up service (localfs) network  .  .  .  .  .  .  .  .  .  ...done
Apr  4 17:14:00 roxeli bluez-coldplug[1330]: ..done
Apr  4 17:14:01 roxeli network-remotefs[1334]: Setting up (remotefs) network interfaces:
Apr  4 17:14:01 roxeli echo[1376]: Starting mail service (Postfix)
Apr  4 17:14:01 roxeli mcelog: mcelog read: No such device
Apr  4 17:14:01 roxeli mcelog[1332]: Starting mcelog... ..done
Apr  4 17:14:01 roxeli nscd[1377]: Starting Name Service Cache Daemon..done
Apr  4 17:14:01 roxeli network-remotefs[1334]: Setting up service (remotefs) network  .  .  .  .  .  .  .  .  .  ...done
Apr  4 17:14:01 roxeli /usr/sbin/cron[1582]: (CRON) INFO (running with inotify support)
Apr  4 17:14:01 roxeli cups[1358]: Starting cupsd..done
Apr  4 17:14:01 roxeli sshd[1333]: Starting SSH daemon..done
Apr  4 17:14:02 roxeli sshd[1675]: Server listening on 0.0.0.0 port 22.
Apr  4 17:14:02 roxeli sshd[1675]: Server listening on :: port 22.
Apr  4 17:14:02 roxeli network[1383]: Shutting down network interfaces:
Apr  4 17:14:02 roxeli network[1383]: eth0      device: Intel Corporation 82545EM Gigabit Ethernet Co
Apr  4 17:14:02 roxeli ifdown[1832]:     eth0      device: Intel Corporation 82545EM Gigabit Ethernet Co
Apr  4 17:14:03 roxeli systemd[1]: PID file /var/run/httpd2.pid not readable (yet?) after start.
Apr  4 17:14:03 roxeli avahi-daemon[729]: Interface eth0.IPv4 no longer relevant for mDNS.
Apr  4 17:14:03 roxeli avahi-daemon[729]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.1.77.
Apr  4 17:14:03 roxeli avahi-daemon[729]: Withdrawing address record for fe80::20c:29ff:fe09:701c on eth0.
Apr  4 17:14:03 roxeli avahi-daemon[729]: Withdrawing address record for 192.168.1.77 on eth0.
Apr  4 17:14:04 roxeli network[1383]: ..doneShutting down service network  .  .  .  .  .  .  .  .  ...done
Apr  4 17:14:04 roxeli systemd[1]: Startup finished in 3s 459ms 469us (kernel) + 16s 100ms 887us (userspace) = 19s 560ms 356us.
Apr  4 17:14:06 roxeli network-remotefs[1592]: Shutting down (remotefs) network interfaces:
Apr  4 17:14:06 roxeli network-remotefs[1592]: Shutting down service (remotefs) network  .  .  .  .  .  .  .  .  ...done

After which a manual ifup eth0 at the console was needed :-(
Comment 39 Marius Tomaschewski 2012-04-04 16:51:54 UTC
Please set DEBUG=EXTRA in /etc/sysconfig/network/config, reboot, modify
boot options to:

noresume systemd.log_level=debug systemd.log_target=kmsg splash=0 showopts

that is splash=silent -> splash=0, resume=... -> noresume, remove the
"quiet" option and add the systemd.* options.

After the system has booted, call:

  tail -500 /var/log/messages > /dev/.sysconfig/network/messages.txt
  dmesg                       > /dev/.sysconfig/network/dmesg.txt
  zypper se -i --detail       > /dev/.sysconfig/network/pkgs.txt

and attach an (tar.bz2) archive of /dev/.sysconfig/network.

See also https://bugzilla.novell.com/show_bug.cgi?id=747931#c12 :-/
Comment 40 Michal Filka 2012-04-05 10:42:47 UTC
I've tested the solution for bnc#732696 and no change. However behaviour is slightly different from above one.

I've just updated system:
> rpm -q sysconfig systemd
sysconfig-0.75.4-2.7.1.x86_64
systemd-37-3.11.1.x86_64

Doesn't work:
Change from DHCP -> static IP (using YaST; network service not running after change)

> /etc/init.d/network status
redirecting to systemctl
network.service - LSB: Configure the localfs depending network interfaces
          Loaded: loaded (/etc/init.d/network)
          Active: inactive (dead) since Thu, 05 Apr 2012 12:35:44 +0200; 8s ago
         Process: 6709 ExecStop=/etc/init.d/network stop (code=exited, status=0/SUCCESS)
         Process: 6099 ExecReload=/etc/init.d/network reload (code=exited, status=0/SUCCESS)
         Process: 5452 ExecStart=/etc/init.d/network start (code=exited, status=0/SUCCESS)                                                                                                                                               
Works:
configured with static IP (doesn't matter if network service runs or nor), after reboot network service runs.
Comment 41 Michal Filka 2012-04-05 10:49:57 UTC
(In reply to comment #38)
> 
> After which a manual ifup eth0 at the console was needed :-(

Just an idea ... do you have samba-client installed? There is an issue related to starting network during boot in such case. bnc#745859
Comment 42 Danny van Delft 2012-04-05 12:32:44 UTC
(In reply to comment #41)
> (In reply to comment #38)
> > 
> > After which a manual ifup eth0 at the console was needed :-(
> 
> Just an idea ... do you have samba-client installed? There is an issue related
> to starting network during boot in such case. bnc#745859

Not at the faulty boot. Prior to this boot, I had done a zypper up. Just prior to that I had removed some unneeded packages, amongst which samba-client (which was not in use anyway, but that seems irrelevant for bnc#745859). So it was installed at boot time when the machine was up and running, then removed, then zypper up-ed, then rebooted which then failed.

However, in accordance to #c39, I have tried to get the output as requested. Since then this (virtual) machine has been rebooted some 120 times, without failing the network setup. Perhaps the extra debugging output "fixes" the problem. I'll try to get the machine into a state where it fails again.
Comment 43 Frederic Crozat 2012-04-05 12:39:56 UTC
could you also try to downgrade to systemd-37-3.8.1   (from upgrade repository) to see if it fixes your issue ?
Comment 44 Danny van Delft 2012-04-05 12:53:20 UTC
(In reply to comment #43)
> could you also try to downgrade to systemd-37-3.8.1   (from upgrade repository)
> to see if it fixes your issue ?

At the moment, with systemd-37-3.11.1, I cannot reproduce, even if it did occur the first time I booted with this version. So, me thinks I should first try to reproduce with systemd-37-3.11.1, before downgrading to systemd-37-3.8.1?
Comment 45 Frederic Crozat 2012-04-05 12:57:08 UTC
yes, of course. Downgrading systemd should be the "last" test :)
Comment 46 Danny van Delft 2012-04-10 10:26:35 UTC
Created attachment 485455 [details]
logging output obtained as described in comment 39

Per request in comment:39

Note at the time this bz2 was made, the eth0 interface had been manually upped. The zypper se tried to contact the download servers, could't due to down network, and hung.

Despite that, glad the machine got into this faulty state at all. The last couple of days the machine had been rebooted some hundreds of times without failing, until this morning...
Comment 47 Marius Tomaschewski 2012-04-12 13:50:05 UTC
(In reply to comment #46)
> Note at the time this bz2 was made, the eth0 interface had been manually upped.

That's ok.

dmesg.txt:
[   54.196701] systemd[1]: network.service changed start -> running
[   54.196710] systemd[1]: Job network.service/start finished, result=done
[   54.196983] systemd[1]: network.target changed dead -> active
[   54.196989] systemd[1]: Job network.target/start finished, result=done
[   54.196997] systemd[1]: remote-fs-pre.target changed dead -> active
[   54.197002] systemd[1]: Job remote-fs-pre.target/start finished, result=done
...
[   54.213945] systemd[1]: About to execute: /etc/init.d/mysql start
[   54.218964] systemd[1]: Forked /etc/init.d/mysql as 1547
[   54.218998] systemd[1]: mysql.service changed dead -> start
...
[   54.484353] systemd[1]: network.service: cgroup is empty
[   54.484396] systemd[1]: About to execute: /etc/init.d/network stop
[   54.489237] systemd[1]: Forked /etc/init.d/network as 1626
[   54.489286] systemd[1]: network.service changed running -> stop

messages.txt:
...
Apr 10 08:40:07 roxeli mysql[1547]: /etc/rc.status: line 57: /dev/stderr: No such device or address
...

This is from /etc/init.d/mysql -> systemct redirection. A bit strange...

Network will be started fine, but then (to resolve deps), systemd is
stopping it (see begin of exdeb.network_1626.1._.stop).

Hmm... for me it looks like a problem with dependencies and there is
the "network.service: cgroup is empty" message before it gets stopped.
Something about mysql.service? Going to install it and take a look.
Comment 48 Per Jessen 2012-04-12 14:08:09 UTC
> messages.txt:
> ...
> Apr 10 08:40:07 roxeli mysql[1547]: /etc/rc.status: line 57: /dev/stderr: No
> such device or address
> ...
> 
> This is from /etc/init.d/mysql -> systemct redirection. A bit strange...

FYI, I see this quite frequently, usually from /etc/init.d/ntp.
Comment 49 Danny van Delft 2012-04-12 14:37:04 UTC
(In reply to comment #47)
> (In reply to comment #46)
...
> ...
> Apr 10 08:40:07 roxeli mysql[1547]: /etc/rc.status: line 57: /dev/stderr: No
> such device or address
> ...
> 
> This is from /etc/init.d/mysql -> systemct redirection. A bit strange...
> 
> Network will be started fine, but then (to resolve deps), systemd is
> stopping it (see begin of exdeb.network_1626.1._.stop).
> 
> Hmm... for me it looks like a problem with dependencies and there is
> the "network.service: cgroup is empty" message before it gets stopped.
> Something about mysql.service? Going to install it and take a look.

See https://bugzilla.novell.com/show_bug.cgi?id=728774 where I reported this phenomenon "way back" ;-)
Comment 50 Stefan Kuehne 2012-04-19 09:15:40 UTC
Same problem here. 
We have around 70 DomUs, mostly with 11.3. 10 DomUs are upgraded to 12.1 and on two we have this problem.
We boot these DomUs every night to backup them. Sometimes this problem occurs. 
Samba is installed on one, Mysql on both of the DomU with network errors.

It's not so easy to reproduce the problem. After a few restarts in succession the problem does not appear again.

I change my /etc/rc.status like in Bug 728774 (previous post) described and post if the problem occurs the next time.
Comment 51 Stefan Kuehne 2012-04-30 06:17:46 UTC
The change of the file /etc/rc.status ( /dev/stderr -> 1>&2 ) seems to help.
The error doesn't occur on the DomU with a changed rc.staus. On a third DomU with no change the error occur 1 time. All DomU where restarted daily. 

I post if this error occur again...
Comment 52 Stefan Kuehne 2012-05-18 11:07:12 UTC
The error occured one time again in the last 3 weeks.
Comment 53 Marius Tomaschewski 2012-05-23 15:31:30 UTC
=> reassign to systemd maintainer to fix RemainAfterExit again,
   see also bug 763533 comment 11.
Comment 54 Frederic Crozat 2012-05-23 15:57:16 UTC
please test systemd package from http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/ (wait until the date there has been updated to May 23 at minimal, package is in rebuilding state atm), it should fix this issue.
Comment 55 H Harun 2012-05-23 17:21:07 UTC
I have the same problem but only on machine with Intel Network card : 82573 Gigabit Ethernet Controller.

Normally I do full installs (select ALL patterns in yast).  When it finishes installing all the rpms, it will reboot and NO network. 
I have tried various configuration mentioned above, it is still not working.  No network.

Then I did a small set of rpm installs.  

Currently small set means these patterns:
Base System
Enhanced Base System
Yast System Administration
Software Management
KDE desktop
KDE base system
X windows
Fonts
Office
Desktop Effects


Guess what, small set of rpms install works. There's NETWORK connection. 
I then add ALL other patterns, reboot and the network is still up.

My guess is some combinations of rpms during the boot is the issue.

...Hazita
Comment 56 Stefan Kuehne 2012-05-24 10:33:25 UTC
i updated the systemd-package ( systemd-37-3.163.1.x86_64.rpm 24-Apr-2012 17:40 )
on two server: 
server1:~ # rpm -qi systemd
Name        : systemd
Version     : 37
Release     : 3.163.1
Architecture: x86_64
Install Date: Thu May 24 12:25:13 2012
Group       : System/Base
Size        : 4198210
License     : GPL-2.0+
Signature   : DSA/SHA1, Tue Apr 24 17:40:16 2012, Key ID 5ace4ca4fd73faf5
Source RPM  : systemd-37-3.163.1.src.rpm
Build Date  : Tue Apr 24 17:39:39 2012
Build Host  : build30
Relocations : (not relocatable)
Vendor      : obs://build.opensuse.org/home:fcrozat
URL         : http://www.freedesktop.org/wiki/Software/systemd


One of these servers will be restarted daily, the other three times a week. I post if the error occur again.

PS: The servers are Xen-DomU. The Dom0 is running OpenSuse 11.3.
Comment 57 Bernhard Wiedemann 2012-05-24 13:00:16 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/122059 Factory / systemd
Comment 58 Frederic Crozat 2012-05-25 11:52:10 UTC
pushed to maintenance team : mr#122366
Comment 59 Bernhard Wiedemann 2012-05-25 13:00:13 UTC
This is an autogenerated message for OBS integration:
This bug (727771) was mentioned in
https://build.opensuse.org/request/show/122369 Factory / systemd
Comment 60 Benjamin Brunner 2012-05-30 13:38:34 UTC
Update released. Resolved fixed.
Comment 61 Swamp Workflow Management 2012-05-30 14:08:35 UTC
openSUSE-RU-2012:0664-1: An update that has four recommended fixes can now be installed.

Category: recommended (low)
Bug References: 727771,746594,746595,747931
CVE References: 
Sources used:
openSUSE 12.1 (src):    systemd-37-3.14.1, systemd-gtk-37-3.14.1
Comment 62 Dr. Werner Fink 2015-07-21 09:39:42 UTC
*** Bug 938773 has been marked as a duplicate of this bug. ***
Comment 63 LTC BugProxy 2015-07-21 09:48:22 UTC
(In reply to LTC BugProxy from comment #4)

> Still the question remains why we only see this on LPAR and not under z/VM.
> Any idea?

Without X-Systemd-RemainAfterExit this (IMHO) should happen on both

*** This bug has been marked as a duplicate of bug 727771 ***
Comment 64 LTC BugProxy 2015-07-21 10:12:44 UTC
Created attachment 641524 [details]
testconf
Comment 65 LTC BugProxy 2015-08-25 12:32:59 UTC
------- Comment From mgrf@de.ibm.com 2015-08-25 12:30 EDT-------
Make previous comment external

The question to the SUSE systemd experts is:

Why is systemd calling "testconf stop" (PID=24525) after the testconf
background process has exited?

Tested on zEC12 LPAR with SLES12.1-Beta2 kernel 3.12.46-2-default

-------------------------------------
(In reply to comment #5)
> Created attachment 100321 [details]
> testconf
>
> Stripped down version of the dumpconf service:
>
> The "start" function of this service script forks itself as a new background
> process with parameters "start background" and then exits.
>
> The background process does the following:
>
>  1) It creates a PID file "/var/run/testconf.pid"
>  2) Sleeps for 2 seconds
>  3) Removes the PID file
>  4) Exits
>
> Installation:
>
>  1) Copy "testconf" init script to /etc/init.d/
>  2) Issue: chkconfig --add testconf
>
> Execution:
>
>  1) Issue: service testconf start
>  2) Inspect /tmp/testconf.log
>
> In the log on LPAR I see the following:
>
> PPID=1 PID=24510: start
> PPID=1 PID=24510:  The activation of testconf is being delayed
> PPID=1 PID=24510: start done
> PPID=1 PID=24510: exit
> PPID=1 PID=24516: start
> PPID=1 PID=24516:  The background process
> PPID=1 PID=24516:  sleep: 0
> PPID=1 PID=24516:  sleep: 1
> PPID=1 PID=24516: start done
> PPID=1 PID=24516: exit
> PPID=1 PID=24525: stop        <--- ????
> PPID=1 PID=24525:  do stop
> PPID=1 PID=24525: stop done
> PPID=1 PID=24525: exit
>
> The question to the SUSE systemd experts is:
>
> Why is systemd calling "testconf stop" (PID=24525) after the testconf
> background process has exited?

Tested on zEC12 LPAR with SLES12.1-Beta2 kernel 3.12.46-2-default

#rpm -qi systemd

Name        : systemd
Version     : 210
Release     : 73.6
Architecture: s390x
Install Date: Tue Aug 25 14:24:37 2015
Group       : System/Base
Size        : 12866072
License     : LGPL-2.1+
Signature   : RSA/SHA256, Wed Aug 19 04:07:47 2015, Key ID 70af9e8139db7c82
Source RPM  : systemd-210-73.6.src.rpm
Build Date  : Wed Aug 19 04:05:37 2015
Build Host  : s390lpd
Relocations : (not relocatable)
Packager    : https://www.suse.com/
Vendor      : SUSE LLC <https://www.suse.com/>
URL         : http://www.freedesktop.org/wiki/Software/systemd
Summary     : A System and Session Manager
Description :
Systemd is a system and service manager, compatible with SysV and LSB
init scripts for Linux. systemd provides aggressive parallelization
capabilities, uses socket and D-Bus activation for starting services,
offers on-demand starting of daemons, keeps track of processes using
Linux cgroups, supports snapshotting and restoring of the system state,
maintains mount and automount points and implements an elaborate
transactional dependency-based service control logic. It can work as a
drop-in replacement for sysvinit.
Distribution: SUSE Linux Enterprise 12

tried testconf service
#cat /tmp/testconf.log

PPID=1 PID=32522: start
PPID=1 PID=32522:  The activation of testconf is being delayed
PPID=1 PID=32522: start done
PPID=1 PID=32522: exit
PPID=1 PID=32528: start
PPID=1 PID=32528:  The background process
PPID=1 PID=32528:  sleep: 0
PPID=1 PID=32528:  sleep: 1
PPID=1 PID=32528: start done
PPID=1 PID=32528: exit

no stop observed.

also

# cat /etc/sysconfig/dumpconf | grep -v "^#" | grep "="
ON_PANIC="dump_reipl"
DELAY_MINUTES="1"
DUMP_TYPE="ccw"
DEVICE="0.0.7e69"
WWPN=""
LUN=""
BOOTPROG="0"
BR_LBA="0"
VMCMD_1=""
VMCMD_2=""
VMCMD_3=""
VMCMD_4=""
VMCMD_5=""
VMCMD_6=""
VMCMD_7=""
VMCMD_8=""

# lsshut
Trigger          Action
========================
Halt             stop
Power off        stop
Reboot           reipl
Restart          dump_reipl
Panic            dump_reipl

# systemctl status dumpconf.service
dumpconf.service - LSB: Configure s390 dump feature
Loaded: loaded (/etc/init.d/dumpconf)
Active: active (exited) since Tue 2015-08-25 11:59:33 MSK; 1min 36s ago
Process: 32741 ExecStop=/etc/init.d/dumpconf stop (code=exited, status=0/SUCCESS)
Process: 32754 ExecStart=/etc/init.d/dumpconf start (code=exited, status=0/SUCCESS)

Aug 25 11:59:33 p23lp33 dumpconf[32754]: The activation of dumpconf is being delayed for 1 minutes
Aug 25 11:59:33 p23lp33 systemd[1]: Started LSB: Configure s390 dump feature.

no stop observed.

# rpm -q systemd s390-tools
systemd-210-73.6.s390x
s390-tools-1.24.1-43.20.s390x