Bug 850732

Summary: Online distribution upgrade breaks network connectivity
Product: [openSUSE] openSUSE 13.1 Reporter: Forgotten User ZC6aJTElLj <forgotten_ZC6aJTElLj>
Component: InstallationAssignee: E-mail List <yast2-maintainers>
Status: RESOLVED NORESPONSE QA Contact: Jiri Srain <jsrain>
Severity: Major    
Priority: P5 - None CC: cwh, mfilka, mike, mt
Version: FinalFlags: cwh: needinfo? (mt)
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE 13.1   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Forgotten User ZC6aJTElLj 2013-11-15 18:43:46 UTC
User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0

When using the method for an online distribution upgrade from 12.3 to 13.1 as specified here:

http://en.opensuse.org/SDB:System_upgrade

after a reboot, the machine is disconnected from the network because the name of the network adapter changes.

I my case, a Realtek NIC that was present as 'eth0' became 'ens3' after reboot, thus the network settings from /etc/sysconfig/network did not apply.

It turned out that the unique hardware IDs under /var/lib/hardware/udi were duplicated. In Yast network settings, the old adapter entry was still shown, but the NIC appeared as new unconfigured item. Maybe this was due to the fact that another driver module is used with the new kernel, I do not know.

The Realtek NIC is very common, and in this specific case was the default adapter in a KVM/QEMU virtual machine, so this will happen very often.

If the naming of the UIDs (or the device names) cannot be retained, at least the instructions for an online distribution upgrade should include a warning, because the whole purpose of an online distribution upgrade is defeated by this bug!


Reproducible: Always

Steps to Reproduce:
1. Do online distribution upgrade as specified here: http://en.opensuse.org/SDB:System_upgrade, e.g. from openSUSE 12.3
2. Reboot system

Actual Results:  
Network connectivity not restored.

Expected Results:  
Network functioning as before the upgrade.
Comment 1 Forgotten User ZC6aJTElLj 2013-11-16 14:06:02 UTC
Oh, I think it is worse - it will happen every time. Obviously, the scheme to name NIC devices in /etc/udev/rules.d/70-persistent-net.rules has changed.

In openSUSE 12.3, the name was assigned based on the PCI bus ID:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:00:19.0", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Whereas in openSUSE 13.1, the name is assigned based on the MAC:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:78:17:b8", ATTR{type}=="1", NAME="eth0"
Comment 2 Marius Tomaschewski 2013-11-18 09:56:40 UTC
The /etc/udev/rules.d/70-persistent-net.rules should still exists after
an upgrade and should be used... Reassign to udev maintainer.
Comment 3 Marius Tomaschewski 2013-11-18 09:59:37 UTC
(In reply to comment #2)
> The /etc/udev/rules.d/70-persistent-net.rules should still exists after
> an upgrade and should be used... Reassign to udev maintainer.

Ahm... more correctly. It IMO shouldn't be rewritten from BUS to MAC.
Comment 4 Robert Milasan 2013-11-18 10:02:53 UTC
No, in 12.3 was the same like in any other version, the rule to name the
network adapter is based on mac address. You can check from 11.2 up to 12.3 all
of them are based on mac address, even SLES.

Not sure what has happened to your machine, but like Marius said, if you do
have the old /etc/udev/rules.d/70-persistent-net.rules rule, everything should
work quite the same.

Please be more specific or attach the rule in question.
Comment 5 Marius Tomaschewski 2013-11-18 10:24:45 UTC
(In reply to comment #4)
> No, in 12.3 was the same like in any other version, the rule to name the
> network adapter is based on mac address.

By default it is MAC, yes. In yast2 network, you can click & change to use
a BUS based rule instead, what the customer did in 12.3...
Comment 6 Robert Milasan 2013-11-18 10:39:44 UTC
Well, then that has nothing to do with udev. If the rules where generated correctly it would work as expected.

The best way is to use the mac address, that doesn't change at all (at least it shouldn't).
Comment 7 Robert Milasan 2013-11-18 10:43:58 UTC
BTW, how do you even generate a rule in yast? I've tried 13.1, but didn't see anything something like that.
Comment 8 Forgotten User ZC6aJTElLj 2013-11-18 11:49:30 UTC
I have looked a little further and found that:

1. There is no generator script /usr/lib/udev/rules.d/75-persistent-net-generator.rules any more in 13.1. Was that one left out on purpose or is it standard with udev 208? Maybe one has to always define the NIC names via "yast network" now? Otherwise, I am at a loss as to where the /etc/udev/rules.d/70-persistent-net.rules comes from.

Under 12.3, there seems to be some special handling for VM and Realtek NICs in the generator script:

# ignore KVM virtual interfaces
ENV{MATCHADDR}=="52:54:00:*", GOTO="persistent_net_generator_end"
# ignore VMWare virtual interfaces
ENV{MATCHADDR}=="00:0c:29:*|00:50:56:*", GOTO="persistent_net_generator_end"
# ignore Hyper-V virtual interfaces
ENV{MATCHADDR}=="00:15:5d:*", GOTO="persistent_net_generator_end"

My VM MACs started with "52:54:00:", so the different naming happens probably only for VM machines.

2. I have now migrated a non-VM machine with an Intel E1000 NIC and everything worked fine with that one.

The rules in question are given in comment #1.
Comment 9 Robert Milasan 2013-11-18 11:57:25 UTC
Virtual network adapters weren't handled prior to 13.1, which means there are no rules for any virtual adapter, because the mac address of the virtual network adapters are random at each boot.

The persistent rule which you have got no idea now it was generated, thus is nothing that I could do from udev side.

Since 13.1 there aren't anymore persistent rules, but predictable rules for naming the network adapters (which imply also virtual network adapters): 
http://www.freedesktop.org/wiki/Software/system/PredictableNetworkInterfaceNames/

So previously with persistent rules, only physical network adapters where handled by udev, now in 13.1 both physical and virtual will be handled by the predictable network naming rules.
Comment 10 Forgotten User ZC6aJTElLj 2013-11-18 13:34:10 UTC
I doubt that VM MACs change at each boot - they are ramdomly created when the VM guest is created. Most probably the NIC name was assigned in the 70-persistent-net.rules when the first configuration by the installation process for 12.3 was done and not by the generator rule.

Under 13.1, one can choose in YaST on how (i.e. based on PCI slot or on MAC) the persistent name should be assigned and a 70-persistent-net.rules file is created (which effectively defeats systemd/udev standard rules).

So, after all, the new scheme MAY give problems depending on how the old persistent rules have been created.

The new NIC naming scheme should at least be listed in the 13.1 release notes and in the online distribution upgrade instructions.
Comment 11 Robert Milasan 2013-11-18 13:48:43 UTC
(In reply to comment #10)
> 
> So, after all, the new scheme MAY give problems depending on how the old
> persistent rules have been created.
> 

No, if using it normally the naming scheme of the persistent rules has not changed in years. If the rule will be or is generated by udev then it will work on any version. The rule which you have in 12.3 has been generated by some other app or manually by you or someone else.

The naming scheme is not the issue, your problem is the way the rule has been create and by what. In previous version of udev there was a script called by 75-persistent-net-generator.rules called write_net_rules which never based itself on BUS_ID or something similar, always on mac address.

> The new NIC naming scheme should at least be listed in the 13.1 release notes
> and in the online distribution upgrade instructions.

I was thinking this has been listed in the release notes. Sorry about that.
Comment 12 Marius Tomaschewski 2013-11-20 22:23:31 UTC
(In reply to comment #6)
> Well, then that has nothing to do with udev. If the rules where generated
> correctly it would work as expected.
> 
> The best way is to use the mac address, that doesn't change at all (at least it
> shouldn't).

This is not completely correct. MAC based matching is unusable for bonding
[it usually a bond sets the MAC of the primary slave on _all_ slaves and
stays with this MAC].
You need BUS based rule as you want to be able to replace the NIC with a
different one [e.g. PCI hotplug] and the new NIC has to get the same name
as the NIC before to be enslaved automatically.

(In reply to comment #7)
> BTW, how do you even generate a rule in yast? I've tried 13.1, but didn't see
> anything something like that.

It were working up to IMO 12.3 -- when not, is IMO a bug as it breaks bonding.

(In reply to comment #11)
> The naming scheme is not the issue, your problem is the way the rule has been
> create and by what. In previous version of udev there was a script called by
> 75-persistent-net-generator.rules called write_net_rules which never based
> itself on BUS_ID or something similar, always on mac address.

Ahm... it were generating MAC based rules for ethernet by default, but
in some cases the rules were AFAIR always BUS based -- e.g. s390, ...
And when there were bonding configured on top of an interface, the rule
has to be rewritten to BUS based one. Otherwise hotplugging does not work.
Comment 13 Marius Tomaschewski 2013-11-20 22:31:34 UTC
(In reply to comment #10)
> I doubt that VM MACs change at each boot - they are ramdomly created when the
> VM guest is created.

And usually while migration / VM clone operations. This were the reason to not assign MAC based rules there. Further, the VM ifnames were usually stable.
When the NIC were full-virtualized (intel, rtl, ...) , the MAC were often not
in the locally administered addresses (0x${mac%%:*} & 0x02) range but in the
range of the manufacturer and there were a rule for -- at least sometimes.
Comment 17 Stefan Hundhammer 2016-03-15 11:30:23 UTC
Marius, see question in comment#15.
Comment 18 Stefan Hundhammer 2016-03-24 14:28:30 UTC
No response for a long time, and this bug is now 860 days old (!) and pretty sure obsolete now anyway.