Bug 747931 - systemd shut off network at boot time in a bridge over vlan configuration
Summary: systemd shut off network at boot time in a bridge over vlan configuration
Status: RESOLVED FIXED
Alias: None
Product: openSUSE 12.1
Classification: openSUSE
Component: Network (show other bugs)
Version: Final
Hardware: Other Other
: P5 - None : Major (vote)
Target Milestone: ---
Assignee: Frederic Crozat
QA Contact: E-mail List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-02-20 20:49 UTC by Forgotten User CxVz4LpaB5
Modified: 2012-05-30 14:09 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Yast logs (4.45 MB, application/x-compressed)
2012-02-20 20:49 UTC, Forgotten User CxVz4LpaB5
Details
DEBUG of network operations (sysconfig) (222.79 KB, application/x-compressed)
2012-02-22 19:51 UTC, Forgotten User CxVz4LpaB5
Details
dom0 debug extra files (327.31 KB, application/x-compressed)
2012-02-24 14:19 UTC, Forgotten User CxVz4LpaB5
Details
sysconfig debug with boot parameters (381.91 KB, application/x-compressed)
2012-03-06 15:54 UTC, Forgotten User CxVz4LpaB5
Details
/dev/.sysconfig/network (23.84 KB, application/x-gzip)
2012-05-24 15:39 UTC, Ralf Müller
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Forgotten User CxVz4LpaB5 2012-02-20 20:49:08 UTC
Created attachment 476952 [details]
Yast logs

User-Agent:       Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0

My configuration looks like this on my XEN system (Linux 3.1.9-1.4-xen #1 SMP Fri Jan 27 08:55:10 UTC 2012 (efb5ff4) x86_64 x86_64 x86_64 GNU/Linux)

    -- vlan 100 -- bridge 3 
    |
eth0|- vlan 200 -- bridge 2 (192.170.2.3/255.255.255.192)
    |
    -- vlan 5   -- bridge 1

When the system restart, it first starts the network and few moments after the network is completely down.

I have also notice this 'bug' on a domu. 

Reproducible: Always

Steps to Reproduce:
1. XEN with latest update
2. configure eth0 -> vlan -> bridge 1
                 |-> vlan2 -> bridge 2 (ip)
                 |-> vlan3 -> bridge 3
3. reboot
Actual Results:  
During the reboot a ping show that the network is activated on the xen server and then shut down. I am afraid to reboot my xen server...

Expected Results:  
systemd should not stop the network
Comment 1 Frederic Crozat 2012-02-21 10:26:03 UTC
please install systemd package http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/ from and sysconfig package from http://download.opensuse.org/repositories/home:/mtomaschewski:/branches:/openSUSE:/12.1:/Update:/Test/openSUSE_12.1/

and retry again, they contains a lot of fixes, including some for network.
Comment 2 Forgotten User CxVz4LpaB5 2012-02-21 18:21:37 UTC
I have tested those packages on a domu that has the same issue, and still have the same problem after a reboot:

Feb 21 12:06:38 napier systemd[1]: Startup finished in 4s 795ms 465us (kernel) + 38s 858ms 816us (userspace) = 43s 654ms 281us.
Feb 21 12:06:39 napier network[1078]: Shutting down network interfaces:
Feb 21 12:06:40 napier network[1078]: eth0      device: Intel Corporation 82540EM Gigabit Ethernet Co
Feb 21 12:06:40 napier ifdown[1617]:     eth0      device: Intel Corporation 82540EM Gigabit Ethernet Co
Feb 21 12:06:45 napier network[1078]: ..doneShutting down service network  .  .  .  .  .  .  .  .  ...done
Comment 3 Marius Tomaschewski 2012-02-22 09:54:10 UTC
Except of the systemd and sysconfig packages, please update all samba
related packages (especially samba-client) from:

http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/

Note: You don't need to _use_ samba client/server -- it is enough, that
the samba-client package is installed (it is by default).

When this does not help,
please set DEBUG=EXTRA in /etc/sysconfig/network/config, reboot and
attach a tar.bz2 of /dev/.sysconfig/network when it occurs again.
Comment 4 Forgotten User CxVz4LpaB5 2012-02-22 19:50:43 UTC
I have check on a domu that has the same issue and there is no samba related packages installed. I have done the DEBUG=EXTRA thing and put the result in a file.
Please that the the issue seems only appear on systems that has been upgraded from 11.4 to 12.1
Comment 5 Forgotten User CxVz4LpaB5 2012-02-22 19:51:58 UTC
Created attachment 477398 [details]
DEBUG of network operations (sysconfig)
Comment 6 Forgotten User CxVz4LpaB5 2012-02-23 16:29:01 UTC
Hi,
Please note that there are no samba related packages in http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/
Comment 7 Frederic Crozat 2012-02-23 16:43:06 UTC
samba packages have been removed from home:fcrozat:systemd repository because they are now available as 12.1 official updates since this morning, please make sure to install them.
Comment 8 Marius Tomaschewski 2012-02-24 10:49:58 UTC
# for i in exdeb.* ; do echo "`head -1 \"$i\"` $i" ; done | cut -b 12- | sort

-> ifup eth0 -o hotplug executed by udev rule
14:35:14.786939917 exdeb.ifup_408.292.1106_.eth0.-o.hotplug

-> network.service started by systemd
14:35:15.318825942 exdeb.network_418.1._.start

-> children of network.service
14:35:20.075431866 exdeb.ifstatus_616.418._.lo
14:35:21.402162843 exdeb.ifup_685.418._.lo.-o.rc.onboot
14:35:24.797735478 exdeb.ifup_791.418._.eth0.-o.rc.onboot
14:35:28.526129276 exdeb.ifstatus_918.418._.eth0.-o.rc.onboot
14:35:31.865805686 exdeb.ifstatus_1014.418._.eth0.-o.rc.onboot

-> network.service were started here and everything were fine.

   But then, systemd (PID 1) executed the "network-remotefs start"
   (as PID 1109) and "network stop" (as PID 1110):

14:35:36.142116474 exdeb.network-remotefs_1109.1._.start
14:35:36.164957394 exdeb.network_1110.1._.stop

-> The "network-remotefs start" made an "exec network start -o remotefs":

14:35:38.196808810 exdeb.network_1109.1._.start.-o.remotefs

   that does not do anything (no remote-fs depending ifcfg's there).

Looks like a dependency problem... I don't see any samba involved.

Romain:

When there are no samba related packages (basically samba-client),
that provide /etc/sysconfig/network/if-up.d/ hooks, you don't need
to update anything.

Please reboot with DEBUG=EXTRA again and set also the boot parameter:

noresume systemd.log_level=debug systemd.log_target=kmsg splash=0 showopts

that is splash=silent -> splash=0, resume=... -> noresume, remove
the "quiet" option and add the systemd.* options.

After the system has booted, call:

  tail -500 /var/log/messages > /dev/.sysconfig/network/messages.txt
  dmesg                       > /dev/.sysconfig/network/dmesg.txt
  zypper se -i --detail       > /dev/.sysconfig/network/pkgs.txt

and attach an archive of /dev/.sysconfig/network again.

BTW: I'm currently using:

# zypper se -i --sort-by-repo --detail | grep -E "systemd|sysconfig"
i | systemd                                  | Paket   | 37-3.153.1                   | i586   | repo-fcrozat     
i | systemd-presets-branding-openSUSE        | Paket   | 0.1.0-32.1                   | noarch | repo-fcrozat     
i | systemd-sysvinit                         | Paket   | 37-3.153.1                   | i586   | repo-fcrozat     
i | sysconfig                                | Paket   | 0.75.4-2.10.test.1           | i586   | repo-mt-12.1     

Hmm... why I've got an i586 kvm machine when installing from url? -> Bug :-)
Comment 9 Forgotten User CxVz4LpaB5 2012-02-24 14:17:30 UTC
Hi,
Oups, I have reboot yesterday before reading your post. I have set the debug=extra in sysconfig/network and here are the results. I will reboot the weekend with those parameters.
But, please not that this time after the reboot (systemd and sysconfig package updated with the rpm from your repo and samba packages updates with latest available packages from the official repo), the network works.

On the domy where I have the same issue and no samba rpms installed, I will test also those extra parameters
Comment 10 Forgotten User CxVz4LpaB5 2012-02-24 14:19:17 UTC
Created attachment 477903 [details]
dom0 debug extra files

On the xen dom0 server where I have the issue, thos are the files resulting after the debug=extra in sysconfig/network
Comment 11 Marius Tomaschewski 2012-02-28 13:00:15 UTC
(In reply to comment #9)
> Hi,
[...]
> with latest available packages from the official repo), the network works.

This sounds good. The logs of the test are in comment 10, right? At least
I don't see any log files that the network would be stopped after start...

When it works, I don't need the systemd logs, that is you don't need to
boot with enabled systemd debug again. Only when it does not work ...

> On the domy where I have the same issue and no samba rpms installed, I will
> test also those extra parameters

OK, thanks for your testing!!
Comment 12 Forgotten User CxVz4LpaB5 2012-03-06 15:54:20 UTC
Created attachment 479774 [details]
sysconfig debug with boot parameters

Here are the systemd, messages, dmesg and network logs.
Please note that the issue reappear even after updating to latest systemd and samba packages...
Comment 13 Marius Tomaschewski 2012-04-04 16:58:25 UTC
(In reply to comment #12)
> Created an attachment (id=479774) [details]
> sysconfig debug with boot parameters
> 
> Here are the systemd, messages, dmesg and network logs.
> Please note that the issue reappear even after updating to latest systemd
> and samba packages...

OK, ... then let's reassign this one to systemd maintainer.
Bug 727771 is also open again :-/
Comment 14 Frederic Crozat 2012-04-05 07:20:19 UTC
which version of systemd package is installed on your system right now ?

could you try to downgrade from one version and see if it works better.
Comment 15 Frederic Crozat 2012-05-23 16:24:46 UTC
please test systemd package from
http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/
(wait until the date there has been updated to May 23 at minimal, package is in
rebuilding state atm), it should fix this issue.
Comment 16 Bernhard Wiedemann 2012-05-24 13:00:36 UTC
This is an autogenerated message for OBS integration:
This bug (747931) was mentioned in
https://build.opensuse.org/request/show/122059 Factory / systemd
Comment 17 Ralf Müller 2012-05-24 15:38:17 UTC
(In reply to comment #15)
> please test systemd package from
> http://download.opensuse.org/repositories/home:/fcrozat:/systemd/openSUSE_12.1/
> (wait until the date there has been updated to May 23 at minimal, package is in
> rebuilding state atm), it should fix this issue.

I have two Tumbleweed DomU's on a relatively slow Xen-Host (Intel(R) Xeon(TM) CPU 3.20GHz). Dom0 is a Tumbleweed install too. I too had the problem of a shut off network after boot. I was able to reproduce this quite reliably when I restarted two DomU's at the same time or when for whatever reason there was a high load in the system.

I installed the above packages. They make the problem occur much less often, but it is still there. Before this patch it was quite sure that at least one of the both rebooted DomU's shut down its network after boot. Now in about one of four reboots, one of the systems shuts down its network. 

/dev/.sysconfig/network tgz attached
Comment 18 Ralf Müller 2012-05-24 15:39:34 UTC
Created attachment 492332 [details]
/dev/.sysconfig/network
Comment 19 Frederic Crozat 2012-05-25 08:48:44 UTC
please make sure you have systemd-37-3.164.1 or later installed.

37-3.163.1 didn't had the patch
Comment 20 Ralf Müller 2012-05-25 09:46:58 UTC
(In reply to comment #19)
> please make sure you have systemd-37-3.164.1 or later installed.

Ok - updated. 10 simultanious reboots of both DomU's done, no problems so far. For me the problem is solved. Thanks a lot.

Can you explain what you have done for the fix?

And when can we expect this update in mainline 12.1/Tumbleweed?
Comment 21 Frederic Crozat 2012-05-25 10:03:20 UTC
In fact, we had fixed the bug sometime ago (making sure network.service is considered a "RemainAfterExit" service, ie when no process are remaining in its cgroup, it won't become dead) but there was a bug in the parsing code I added in systemd to process the "X-Systemd-RemainAfterExit" LSB header (in /etc/init.d/network ) which was ignored when added after Description: header.

Let's push this fix as maintenance update !
Comment 22 Frederic Crozat 2012-05-25 11:50:30 UTC
pushed to maintenance team : mr#122366
Comment 23 Benjamin Brunner 2012-05-25 12:33:51 UTC
Thanks for the submission. I started an update for 12.1.
Comment 24 Bernhard Wiedemann 2012-05-25 13:00:21 UTC
This is an autogenerated message for OBS integration:
This bug (747931) was mentioned in
https://build.opensuse.org/request/show/122369 Factory / systemd
Comment 25 Benjamin Brunner 2012-05-30 13:38:40 UTC
Update released. Resolved fixed.
Comment 26 Swamp Workflow Management 2012-05-30 14:09:14 UTC
openSUSE-RU-2012:0664-1: An update that has four recommended fixes can now be installed.

Category: recommended (low)
Bug References: 727771,746594,746595,747931
CVE References: 
Sources used:
openSUSE 12.1 (src):    systemd-37-3.14.1, systemd-gtk-37-3.14.1