Bug 803058

Summary: No network after upgrading to RC1
Product: [openSUSE] openSUSE 12.3 Reporter: Forgotten User eDPGYP6_cn <forgotten_eDPGYP6_cn>
Component: NetworkAssignee: openSUSE GNOME <os.gnome.maintainers>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P5 - None CC: aplanas, fcrozat, freek, Greg.Freemyer, jweberhofer, koenig, lmuelle, lnussel, mfilka, mt, o.nicolas, peter, rjschwei, simon.palmer
Version: RC 1Flags: coolo: SHIP_STOPPER+
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: Yes
Marketing QA Status: --- IT Deployment: ---
Attachments: migrate NETWORKMANAGER=no setting

Description Forgotten User eDPGYP6_cn 2013-02-11 12:02:50 UTC 
I was using a factory version. Today I decided to upgrade to 12.3 and so did a zypper dup with the iso openSUSE-12.3-DVD-Build0024=x86_64

Ever since that the network is not coming up at all.

I went to YaST network and saw that the "Traditional" was selected even though I have used NetworkManager all along prior to the update. I changed it to netowrkmanager manually.

If I do a `rcnetwork status` I get:

==========

Checking service network . . . dead
redirecting to "systemctl status network.service"
network.service - LSB: Configure network interfaces and set up rotuing
LoadeD: loaded (/etc/init.d/network)
Active: failed (Result: exit-code) since <<time>>

dhcpd[6878]: eth0: broadcasting for a lease
dhcpd[6878]: eth0: timed out

============




If I do a `rcnetwork restart` I get:

============
redirecting to systemctl restart network.service"
Job for network.service failed. See 'systemctl status network.service' and 'journalctl -n' for details.

==============


I feel this is a blocker because if I have made this update in my home pc where I wont have any other computer in the same network, it would have made a fix impossible to have. Feel free to remove the SHIP_STOPPER if it is not so critical.
Comment 1 peter czanik 2013-02-14 07:07:53 UTC 
Also experienced while updating from beta1 to rc1. Here it had the side effect that when I connected to a wifi network instead of Gbit Ethernet, the default route was set to the IP of the gw used with the wired network, even when it was not connected. I used "rm /var/lib/dhcp*/*" as a workaround as I did not notice the NM->traditional change...
Comment 2 Forgotten User eDPGYP6_cn 2013-02-14 10:00:39 UTC 
Removing the /var/lib/dhcp*/* didnt fix the network issue :(

Since this is my primary work laptop, I couldnt take the downtime longer. I went and reinstalled (as per suggestions in some fedora forums) and that has fixed the issue. So I do not have the setup with me now. After the re-installation network access is fine. From the fedora forum, it appeared to me that this is a systemd related issue.
Comment 3 Freek de Kruijf 2013-02-14 11:23:49 UTC 
This is similar to bug #803008, see workaround there.
Comment 4 Marius Tomaschewski 2013-02-18 14:07:12 UTC 
sysconfig and most other packages has been changed to not use NETWORKMANAGER
variable any more, but a network.service alias link.

See also release notes bug #800771 and - still unfixed - bug #798348, to
adopt switching between NetworkManager and /etc/init.d/network in yast2.
[/etc/init.d/network is currently active only while miminal installation,
 in all other cases NetworkManager because yast2 is unable to switch it
 as needed.]

To enable NetworkManager, use:

  systemctl stop network.service
  systemctl --force enable NetworkManager.service
  systemctl start network.service

to disable NetworkManager, use:

  systemctl stop network.service
  systemctl --force disable NetworkManager.service
  systemctl start network.service

This command:
   systemctl --no-pager -p Id show network.service

reports "Id=NetworkManager.service" when NetworkManager acts as the network
service, "Id=network.service" means /etc/init.d/network.

Please use above commands to enable/disable NetworkManager as needed and
reboot. Does it work as expected after?

When not, and you are using /etc/init.d/network, please call "supportconfig"
(supportutils package) and attach the archive it creates.
Comment 5 Forgotten User eDPGYP6_cn 2013-02-18 14:48:46 UTC 
As mentioned in my previous comment, I do not have the setup anymore. I reinstalled and the issue vanished. If the above commands will resolve the issue, is it not possible to execute it automatically on the first boot after the `zypper dup` ?
Comment 6 Marius Tomaschewski 2013-02-19 14:54:08 UTC 
Created attachment 525267 [details]
migrate NETWORKMANAGER=no setting

(In reply to comment #5)
> is it not possible to execute it automatically on the first boot after
> the `zypper dup` ?

Well, yes -- we have to fix it.

The patch is a result of:

osc rdiff openSUSE:Factory/sysconfig home:mtomaschewski:tests/sysconfig

It adds a migration hook disabling NetworkManager.service on update
from an installation with a NETWORKMANAGER=no setting.

That is basically a:

       if [ "X$NETWORKMANAGER" = "Xno" -a -x /bin/systemctl ] ; then
               /bin/systemctl --force disable NetworkManager.service || :
       fi

plus guard to make it only on update from system using NETWORKMANAGER.

My "zypper dup" is still running, that is, I expect we need more than
this, e.g.

  /usr/lib/systemd/system-preset/default-openSUSE.preset

currently contains an "enable NetworkManager.service" line ...
Comment 7 Marius Tomaschewski 2013-02-19 15:00:31 UTC 
Frederic, Ludwig,

while NetworkManager.service on <=12.2 were using the NETWORKMANAGER
variable and the service file were always enabled, it isn't / should't
be always enabled any more -> it breaks update.

So we have to remove it from the preset file, right?
Comment 8 Marius Tomaschewski 2013-02-19 15:06:33 UTC 
*** Bug 802879 has been marked as a duplicate of this bug. ***
Comment 9 Frederic Crozat 2013-02-19 15:08:26 UTC 
(In reply to comment #7)
> Frederic, Ludwig,
> 
> while NetworkManager.service on <=12.2 were using the NETWORKMANAGER
> variable and the service file were always enabled, it isn't / should't
> be always enabled any more -> it breaks update.
> 
> So we have to remove it from the preset file, right?

Which package will take care of calling "systemctl enable network.service /
systemctl enable NetworkManager.service" ?  If we are sure packages will enable
the services (which will create the needed symlink), I'm fine to remove
NetworkManager.service from the preset.
Comment 10 Marius Tomaschewski 2013-02-19 15:16:43 UTC 
(In reply to comment #9)
> (In reply to comment #7)
> > Frederic, Ludwig,
> > 
> > while NetworkManager.service on <=12.2 were using the NETWORKMANAGER
> > variable and the service file were always enabled, it isn't / should't
> > be always enabled any more -> it breaks update.
> > 
> > So we have to remove it from the preset file, right?
> 
> Which package will take care of calling "systemctl enable network.service /
> systemctl enable NetworkManager.service" ?  If we are sure packages will enable
> the services (which will create the needed symlink), I'm fine to remove
> NetworkManager.service from the preset.

On a fresh install this is job of yast2 according to the choice in the proposal.
Comment 11 Marius Tomaschewski 2013-02-19 15:21:42 UTC 
The "systemctl enable network.service" alias %{fillup_and_insserv -fY network}
that activates /etc/init.d/network is done in sysconfig.spec file as before.

And it is IM fine when /etc/init.d/network remains active, as we need only
either "enable" or "disable" of NetworkManager.service, which masks the
network.service provided via /etc/init.d/network (when NM gets enabled).
Comment 12 Marius Tomaschewski 2013-02-19 17:01:22 UTC 
I've tested with NETWORKMANAGER=yes and noticed, that we seem to need
an explicit "enable" as well, otherwise the NetworkManager.service is
enabled after "zypper dup", but the network.service link is missed.

Going to retest using this code:

        case $NETWORKMANAGER in
                # explicitly disable, so /etc/init.d/network is used
                no)     ACTION=disable ;;
                # explicitly enable to create network.service alias
                yes)    ACTION=enable  ;;
                *)      ACTION=""      ;;
        esac
        if [ -n "$ACTION" -x /bin/systemctl ] ; then
                /bin/systemctl --force $ACTION NetworkManager.service || :
        fi
Comment 13 Marius Tomaschewski 2013-02-19 17:36:38 UTC 
Ahm... no, the above will IMO not work too. The enable has to be done in
%post of the _new_ NetworkManager.spec.
Comment 14 Marius Tomaschewski 2013-02-19 18:42:40 UTC 
OK, there are new sysconfig & NetworkManager packages at:

http://download.opensuse.org/repositories/home:/mtomaschewski:/tests/openSUSE_Factory/

I'm currently testing it....

IMO the disable has to be called early, so I keep it in sysconfig
%post. But the "enable" has to be done in the NetworkManager %post
_after_ the installation of the NetworkManager.service file which
provides the alias link:

sysconfig %pre:
	# trigger update from < 0.80.x only (-> openSUSE 12.3),
	# (guard for variable added to network/config later...)
	if grep -qs "^NETWORKMANAGER=" \
		var/adm/fillup-templates/sysconfig.config-network ; then
		grep -s "^NETWORKMANAGER=" etc/sysconfig/network/config \
			> /dev/.sysconfig/network/migrate_nm_variable
	fi

sysconfig %post:
if [ -f /dev/.sysconfig/network/migrate_nm_variable ] ; then
	. /dev/.sysconfig/network/migrate_nm_variable
	if [ -x /bin/systemctl -a -x /usr/sbin/NetworkManager ] ; then
		case $NETWORKMANAGER in
		# bnc#803058:
		# explicitly disable, so /etc/init.d/network is not masked.
		# The enablement is done in the NetworkManager post script
		# to create the network.service alias link as needed.
		no) /bin/systemctl --force disable NetworkManager.service || : ;;
		esac
	fi
	# keep the /dev/.sysconfig/network/migrate_nm_variable file,
	# so the NetworkManager post-install is able to enable later.
fi

NetworkManager %post:
# sysconfig creates this file while update from openSUSE <= 12.2,
# which were using the NETWORKMANAGER=yes/no sysconfig variable.
# When it exists, enable/disable our service accordingly, so the
# network.service alias link refers to the same service as before.
if [ -f /dev/.sysconfig/network/migrate_nm_variable ] ; then
	. /dev/.sysconfig/network/migrate_nm_variable
	rm -f /dev/.sysconfig/network/migrate_nm_variable
	if [ -x /bin/systemctl ] ; then
		case $NETWORKMANAGER in
		no)  /bin/systemctl --force disable NetworkManager.service || : ;;
		yes) /bin/systemctl --force  enable NetworkManager.service || : ;;
		esac
	fi
fi
Comment 15 Marius Tomaschewski 2013-02-20 17:20:54 UTC 
*** Bug 804577 has been marked as a duplicate of this bug. ***
Comment 16 Marius Tomaschewski 2013-02-20 18:08:43 UTC 
OK, the packages in:

http://download.opensuse.org/repositories/home:/mtomaschewski:/tests/openSUSE_Factory/

basically work fine in NM=no case. It almost works also in NM=yes case,
but the restart_on_update NM does fails / timeouts.

Frederic, any idea how to solve it?


Before "zypper dup":

trinity9:~ # systemctl status network.service
network.service - LSB: Configure the localfs depending network interfaces
          Loaded: loaded (/etc/init.d/network)
          Active: active (exited) since Wed, 20 Feb 2013 18:04:49 +0100; 17min ago
          CGroup: name=systemd:/system/network.service

trinity9:~ # systemctl status NetworkManager.service
NetworkManager.service - Network Manager
          Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled)
          Active: active (running) since Wed, 20 Feb 2013 18:06:15 +0100; 15min ago
         Process: 1866 ExecStartPre=/usr/bin/test x${NETWORKMANAGER} = xyes (code=exited, status=0/SUCCESS)
        Main PID: 1869 (NetworkManager)
          CGroup: name=systemd:/system/NetworkManager.service
                  ├ 1869 /usr/sbin/NetworkManager --no-daemon
                  ├ 2116 /sbin/dhclient -d -4 -sf /usr/lib/nm-dhcp-client.action -pf /var/run/d...
                  └ 2118 /sbin/dhclient -d -6 -N -sf /usr/lib/nm-dhcp-client.action -pf /var/ru...


rpm --scripts traces:

*** sysconfig:
+ '[' -f /dev/.sysconfig/network/migrate_nm_variable -a -x /bin/systemctl ']'
+ . /dev/.sysconfig/network/migrate_nm_variable
++ NETWORKMANAGER=yes
+ case $NETWORKMANAGER in
+ ln -sf /usr/lib/systemd/system/NetworkManager.service /etc/systemd/system/network.service

*** NetworkManager:

+ '[' -f /dev/.sysconfig/network/migrate_nm_variable ']'
+ . /dev/.sysconfig/network/migrate_nm_variable
++ NETWORKMANAGER=yes
+ '[' Xyes = Xyes ']'
+ /usr/bin/systemctl --force enable NetworkManager.service
+ '[' '!' -e /etc/systemd/system/network.service ']'
+ rm -f /dev/.sysconfig/network/migrate_nm_variable
+ test -n ''
+ FIRST_ARG=2
+ '[' -d /var/lib/systemd/migrated ']'
+ for service in NetworkManager.service
+ sysv_service=NetworkManager
+ '[' '!' -e /var/lib/systemd/migrated/NetworkManager ']'
+ /usr/bin/systemctl daemon-reload
+ '[' -n '' ']'
+ '[' 2 -eq 1 ']'
+ test -n ''
+ FIRST_ARG=1
+ '[' 1 -ge 1 ']'
+ /bin/systemctl try-restart NetworkManager.service
+ :


Status after the "zypper dup" run [network is up, NM tasks killed]:

trinity9:~ # systemctl status network.service ; systemctl status NetworkManager.service  ; ls -l /etc/systemd/system/ ; systemctl --no-pager -p Id show network.service
NetworkManager.service - Network Manager
	  Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled)
	  Active: failed (Result: timeout) since Wed, 2013-02-20 18:39:14 CET; 11min ago
	Main PID: 16061
	  CGroup: name=systemd:/system/NetworkManager.service

Feb 20 18:34:44 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> NetworkManage...
Feb 20 18:34:44 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> Read config f...
Feb 20 18:34:44 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> WEXT support ...
Feb 20 18:36:14 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> caught signal...
NetworkManager.service - Network Manager
	  Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled)
	  Active: failed (Result: timeout) since Wed, 2013-02-20 18:39:14 CET; 11min ago
	Main PID: 16061
	  CGroup: name=systemd:/system/NetworkManager.service

Feb 20 18:34:44 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> NetworkManage...
Feb 20 18:34:44 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> Read config f...
Feb 20 18:34:44 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> WEXT support ...
Feb 20 18:36:14 trinity9.lab31.mat-home.net NetworkManager[16061]: <info> caught signal...
insgesamt 16
lrwxrwxrwx 1 root root   40 20. Feb 18:25 default.target -> /usr/lib/systemd/system/runlevel3.target
drwxr-xr-x 2 root root 4096 20. Feb 18:25 default.target.wants
drwxr-xr-x 2 root root 4096 20. Feb 18:25 getty.target.wants
lrwxrwxrwx 1 root root    9 20. Feb 18:04 klogd.service -> /dev/null
drwxr-xr-x 2 root root 4096 20. Feb 18:27 multi-user.target.wants
lrwxrwxrwx 1 root root   46 20. Feb 18:25 network.service -> /usr/lib/systemd/system/NetworkManager.service
drwxr-xr-x 2 root root 4096 20. Feb 09:04 network.target.wants
lrwxrwxrwx 1 root root   39 20. Feb 18:27 syslog.service -> /usr/lib/systemd/system/rsyslog.service
Id=NetworkManager.service
Comment 17 Marius Tomaschewski 2013-02-20 18:39:34 UTC 
FYI:
I've changed to use "systemctl mask network.service" while NM=yes again in
most recent sysconfig rpms instead of link to not yet existing NM service.
Before, it wasn't helpful, but as enable seems to be NM-enable to create
the alias now [so there was something wrong] ... I'll retest it tomorrow.
Comment 18 Frederic Crozat 2013-02-21 09:36:23 UTC 
would probably need a more complete log of NM (journalctl -b -u NetworkManager.service) to really understand what is going on..
Comment 19 Marius Tomaschewski 2013-02-21 13:16:47 UTC 
(In reply to comment #18)
> would probably need a more complete log of NM (journalctl -b -u
> NetworkManager.service) to really understand what is going on..

There is IMO nothing special visible in the NM logs [here the part
without to filter with -u to make additional logs visible]:

Feb 21 13:15:47 trinity9.lab31.mat-home.net dbus-daemon[453]: dbus[453]: [system] Reloaded configuration
Feb 21 13:15:47 trinity9.lab31.mat-home.net dbus[453]: [system] Reloaded configuration
Feb 21 13:15:47 trinity9.lab31.mat-home.net systemd[1]: Reloading.
...
Feb 21 13:15:47 trinity9.lab31.mat-home.net NetworkManager[461]: <info> caught signal 15, shutting down normally.
Feb 21 13:15:47 trinity9.lab31.mat-home.net NetworkManager[461]: <warn> quit request received, terminating...
Feb 21 13:15:47 trinity9.lab31.mat-home.net NetworkManager[461]: <info> Clearing nscd hosts cache.
Feb 21 13:15:47 trinity9.lab31.mat-home.net NetworkManager[461]: <info> exiting (success)
Feb 21 13:17:17 trinity9.lab31.mat-home.net systemd[1]: NetworkManager.service stopping timed out (2). Killing.
Feb 21 13:17:17 trinity9.lab31.mat-home.net systemd[1]: Unit NetworkManager.service entered failed state.
Feb 21 13:17:17 trinity9.lab31.mat-home.net NetworkManager[14893]: <info> NetworkManager (version 0.9.6.4) is starting...
Feb 21 13:17:17 trinity9.lab31.mat-home.net NetworkManager[14893]: <info> Read config file /etc/NetworkManager/NetworkManager.conf
Feb 21 13:17:17 trinity9.lab31.mat-home.net NetworkManager[14893]: <info> WEXT support is enabled
Feb 21 13:18:47 trinity9.lab31.mat-home.net systemd[1]: NetworkManager.service operation timed out. Terminating.
Feb 21 13:18:47 trinity9.lab31.mat-home.net NetworkManager[14893]: <info> caught signal 15, shutting down normally.
Feb 21 13:20:17 trinity9.lab31.mat-home.net systemd[1]: NetworkManager.service stopping timed out (2). Killing.
Feb 21 13:21:47 trinity9.lab31.mat-home.net systemd[1]: NetworkManager.service still around after SIGKILL (2). Entering failed mode.
Feb 21 13:21:47 trinity9.lab31.mat-home.net systemd[1]: Job NetworkManager-wait-online.service/start failed with result 'dependency'.
Feb 21 13:21:47 trinity9.lab31.mat-home.net systemd[1]: Unit NetworkManager.service entered failed state.
Feb 21 13:21:53 trinity9.lab31.mat-home.net systemd[1]: Reloading.

More interesting seems to be the "Warning: Unit file changed on disk"
I've seen via ssh while try-restart were running...
Comment 24 Marius Tomaschewski 2013-02-21 14:36:05 UTC 
Created attachment 525768 [details]
Capture/logs from NETWORKMANAGER=yes test [try-restart problems] #3
Comment 30 Marius Tomaschewski 2013-02-22 08:47:21 UTC 
OK,
- requested submission of sysconfig to openSUSE:Factory in 156054 and
- NetworkManager -> GNOME:Factory in 156055 for review by NM maintainers.

+ Reassign to NetworkManager for review.
Comment 31 Bernhard Wiedemann 2013-02-22 09:00:10 UTC 
This is an autogenerated message for OBS integration:
This bug (803058) was mentioned in
https://build.opensuse.org/request/show/156054 Factory / sysconfig
Comment 32 Marius Tomaschewski 2013-02-22 09:09:48 UTC 
Removed "enable NetworkManager.service" from presets in request
156065 systemd-presets-branding-openSUSE  -> Base:System

- Do not unconditionally enable NetworkManager service as this
  breaks running /etc/init.d/network on update (bnc#803058).
[While NetworkManager were using NETWORKMANAGER variable and the
 service always enabled on < 12.3, it is not enabled on 12.3 any
 more when /etc/init.d/network is used]
Comment 33 Bernhard Wiedemann 2013-02-22 10:00:09 UTC 
This is an autogenerated message for OBS integration:
This bug (803058) was mentioned in
https://build.opensuse.org/request/show/156066 Factory / systemd-presets-branding-openSUSE
Comment 34 Marius Tomaschewski 2013-02-26 17:06:57 UTC 
*** Bug 802473 has been marked as a duplicate of this bug. ***
Comment 35 Bernhard Wiedemann 2013-02-27 16:00:23 UTC 
This is an autogenerated message for OBS integration:
This bug (803058) was mentioned in
https://build.opensuse.org/request/show/156714 Factory / NetworkManager
Comment 36 Greg Freemyer 2013-03-01 23:11:27 UTC 
Is this fixed?  I had the bug in RC1, but with a recent zypper dup, all is well for me.
Comment 37 Stephan Kulow 2013-03-03 07:39:20 UTC 
looks like it, 156714 is in 12.3
Comment 38 Marius Tomaschewski 2013-03-04 11:28:56 UTC 
Note: The fix does not catch work for 12.3-RC1 -> 12.3-RC2 updates,
but only for 12.2 -> 12.3 updates: it checks if the old sysconfig
package were still using the NETWORKMANAGER variable or not.

The fixed packages are: http://download.opensuse.org/factory-snapshot/
(sysconfig-0.80.5, NetworkManager, systemd-presets-branding-openSUSE)
Comment 39 Marius Tomaschewski 2013-03-04 11:29:06 UTC 
*** Bug 802761 has been marked as a duplicate of this bug. ***
Comment 40 Alberto Planas Dominguez 2013-03-04 12:34:12 UTC 
I am upgrading 12.1 and 12.2 to 12.3 using the RC2 NET image. Both upgrading fails in sysconfig (sysconfig-0.80.5-1.1.1.x86_64.rpm)

line 11 /dev/.sysconfig/network/migrate_nm_variable: No such file or directory
Comment 41 Marius Tomaschewski 2013-03-04 14:10:59 UTC 
(In reply to comment #40)
> I am upgrading 12.1 and 12.2 to 12.3 using the RC2 NET image. Both upgrading
> fails in sysconfig (sysconfig-0.80.5-1.1.1.x86_64.rpm)
> 
> line 11 /dev/.sysconfig/network/migrate_nm_variable: No such file or directory

Yes, see bug#806989 -- there is a mkdir -p missed in ISO boot + update case :-(
Comment 42 Marius Tomaschewski 2013-03-18 15:38:20 UTC 
*** Bug 809233 has been marked as a duplicate of this bug. ***
Comment 43 Simon Palmer 2013-03-23 00:43:10 UTC 
Is this the same bug:
I upgraded from opensuse12.2 to 12.3, and found my network didn't work.
I have an eth0 with 0.0.0.0/32, and a bridge br0 with a valid static IP.
I've never use NM on this host

After the upgrade, even though my network in yast and in /etc/sysconfig/network/ifcfg-eth0 said eth0 should have 0.0.0.0 for the bridge, it was picking up a DHCP from my router, giving me 2 IP's on the same subnet, and no comms.

I did this (from further up) to fix:
systemctl stop network.service
systemctl --force disable NetworkManager.service
systemctl start network.service

and not had the problem since.
Si
Comment 44 Swamp Workflow Management 2019-05-20 10:30:16 UTC 
This is an autogenerated message for OBS integration:
This bug (803058) was mentioned in
https://build.opensuse.org/request/show/704198 Factory / NetworkManager
Comment 45 Swamp Workflow Management 2019-05-21 11:20:10 UTC 
This is an autogenerated message for OBS integration:
This bug (803058) was mentioned in
https://build.opensuse.org/request/show/704415 Factory / NetworkManager