|
Bugzilla – Full Text Bug Listing |
| Summary: | Unusual network config - enp13s0 and enp14s0, but one is called eth1 | ||
|---|---|---|---|
| Product: | [openSUSE] openSUSE Tumbleweed | Reporter: | Per Jessen <per> |
| Component: | Network | Assignee: | Robert Milasan <rmilasan> |
| Status: | RESOLVED FIXED | QA Contact: | E-mail List <qa-bugs> |
| Severity: | Normal | ||
| Priority: | P4 - Low | CC: | lduncan, mmarek, mt, ohering, werner |
| Version: | 13.1 Beta 1 | ||
| Target Milestone: | --- | ||
| Hardware: | Other | ||
| OS: | Other | ||
| Whiteboard: | |||
| Found By: | --- | Services Priority: | |
| Business Priority: | Blocker: | --- | |
| Marketing QA Status: | --- | IT Deployment: | --- |
| Attachments: |
udevadm test
60-persistent-input.rules from the initrd mkinitrd-support-predictable-names-at-boot.patch |
||
(In reply to comment #0) > > Where did that 'eth1' come from?? 'eth1' is the interface for the iscsi root device. > Also, for some reason both interfaces are configured with BOOTPROTO='dhcp' > and STARTMODE='nfsroot', although I only configured one during installation. Probably related to bug#820408 y2logs are in attachment#539810 [details] Ping? I have no idea why I was assigned. The actions of the fellow Chinese colleagues often do not make sense to me. As for the issue at hand: I would make a guess that there is no entry in /etc/udev/rules.d/70-persistent-net.rules that matches the MAC address of eth1 (00:30:48:90:80:77). Well, that would be for 12.3. For factory and since systemd-197, it should be the other way around: systemd-197 introduced a new logic, described at http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/, and which has already created a sizable mailing list thread on [opensuse-factory]. Assuming that the eth->en rename works fine in systemd/udev, that would mean that you DO have an entry for 00:30:48:90:80:77 in 70-persistent-net.rules that should be removed. Assigning to systemd/udev package owner. fcrozat may have more ideas. Device renaming requires that the device not be in the UP state. However, iSCSI may be pinning the device, though I have not researched in the kernel code whether that is what happens. So, if you have root on iSCSI, then any interface renaming on the supposedly-pinned interface needs to occur before iSCSI starts, i.e. in the initrd. I suppose the udev renaming rule files are not present in the initrd. (In reply to comment #3, comment#4) > I would make a guess that there is no entry in > /etc/udev/rules.d/70-persistent-net.rules that matches the MAC address of eth1 > (00:30:48:90:80:77). Well, that would be for 12.3. This was a fresh Factory install, Milestone 1. There should not have been a "70-persistent-net.rules", I think. > Device renaming requires that the device not be in the UP state. > So, if you have root on iSCSI, then any interface renaming on the > supposedly-pinned interface needs to occur before iSCSI starts, i.e. in the > initrd. I suppose the udev renaming rule files are not present in the initrd. Okay, interesting. See also recently opened report, bug#842450. Can you check if "70-persistent-net.rules" is empty? (In reply to comment #6) > Can you check if "70-persistent-net.rules" is empty? The system was 13.1M1, it's long gone, but I'm pretty certain there was no "70-persistent-net.rules". With 13.1B1 there isn't one. Per, can you run: udevadm test /sys/class/net/eth1 and attach the output to the bug. Err, not really - as I said, that system is long gone. However, I can probably get another identical box installed with 13.1B1, and run it there. Well please do, without seeing whats going on, I can't do anything, as I can't reproduce this. Created attachment 560981 [details]
udevadm test
Looks like Jan was spot on earlier:
changing net interface name from 'eth1' to 'enp14s0'
error changing net interface name eth1 to enp14s0: Device or resource busy
*** Bug 842450 has been marked as a duplicate of this bug. *** This usually happen when we already have the name set to eth1 in out case and ifup already as ran for eth1, meaning that ifup eth1 happen and only before we are trying to rename, ending up in that error. This could be an issue with 77-network.rules. I would suggest to rename /usr/lib/udev/rules.d/77-network.rules to /usr/lib/udev/rules.d/86-network.rules, reboot and try again udevadm test ... or check if eth1 has been renamed. Or rename /usr/lib/udev/rules.d/80-net-name-slot.rules to /usr/lib/udev/rules.d/76-net-name-slot.rules and of course reboot and check if the network has been renamed. Actually please try both, as I can't reproduce the issue, I would like to see the behavior. (In reply to comment #13) > This usually happen when we already have the name set to eth1 in out case and > ifup already as ran for eth1, meaning that ifup eth1 happen and only before we > are trying to rename, ending up in that error. > > This could be an issue with 77-network.rules. > > I would suggest to rename /usr/lib/udev/rules.d/77-network.rules to > /usr/lib/udev/rules.d/86-network.rules, reboot and try again udevadm test ... > or check if eth1 has been renamed. Okay, I tried that - eth1 was not renamed, and the udevadm test was identical to the previous one. > Or rename /usr/lib/udev/rules.d/80-net-name-slot.rules to > /usr/lib/udev/rules.d/76-net-name-slot.rules and of course reboot and check if > the network has been renamed. Okay, I tried that - eth1 was not renamed. udevadm test was also identical. Hmm ok, will try to look into it, but still not possibility of reproduce the issue. Can you please tell me what machine/network adapters are you using? (In reply to comment #13) > I would suggest to rename /usr/lib/udev/rules.d/77-network.rules to > /usr/lib/udev/rules.d/86-network.rules, sysconfig installs a 77-network.rules since ages (9.x?), that is when the 80-net-name-slot.rules renames, it is already too late. I'd prefer to not rename 77-network.rules as it is mentioned in diverse docs ... (In reply to comment #14) > Or rename /usr/lib/udev/rules.d/80-net-name-slot.rules to > /usr/lib/udev/rules.d/76-net-name-slot.rules and of course reboot > and check if the network has been renamed. Yes, better. Don't forget to call "mkinitrd" to update the rules in the initrd (does mkinitrd [dracul?] copy the rules to the initrd?). On iSCSI-root the rename is possible only in the initrd. The system cannot change it any more. So "udevadm test" will show it, but to test if the rename realy works, a reboot is required. (In reply to comment #16) > Hmm ok, will try to look into it, but still not possibility of reproduce the > issue. > > Can you please tell me what machine/network adapters are you using? It is a Supermicro PDSML-LN2+ board, it has two onboard GigE interfaces, one Intel 82573E, one Intel 82573L. (In reply to comment #17) > (In reply to comment #14) > > Or rename /usr/lib/udev/rules.d/80-net-name-slot.rules to > > /usr/lib/udev/rules.d/76-net-name-slot.rules and of course reboot > > and check if the network has been renamed. > > Yes, better. > > Don't forget to call "mkinitrd" to update the rules in the initrd > (does mkinitrd [dracul?] copy the rules to the initrd?). Okay, that broke the system :-) although it boots from the iscsi drive, it cannot complete the boot-up due to not getting access to the root filesystem. I'll have to get a serial console on this to see what's happening. (In reply to comment #19) > (In reply to comment #17) > > > (In reply to comment #14) > > > Or rename /usr/lib/udev/rules.d/80-net-name-slot.rules to > > > /usr/lib/udev/rules.d/76-net-name-slot.rules and of course reboot > > > and check if the network has been renamed. > > > > Yes, better. > > > > Don't forget to call "mkinitrd" to update the rules in the initrd > > (does mkinitrd [dracul?] copy the rules to the initrd?). > > Okay, that broke the system :-) although it boots from the iscsi drive, it > cannot complete the boot-up due to not getting access to the root filesystem. > I'll have to get a serial console on this to see what's happening. Well, the problem seems to be that the faulty network config (see bug#843451) was copied into the initrd, which causes the startup to run dhcp to get an address for eth0/enp14s0. This times out, but there is no attempt to try dhcp with eth1/enp13s0. I'll try to fix the initrd with a working network config. I have now tried a few different things, each time leading to a non-bootable system. In my last attempt, I ran "mkinitrd -f network -D eth1", which finally created a working initrd. The network setup hasn't changed though:
# ip addr
[snip]
2: enp13s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:30:48:92:f3:14 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:30:48:92:f3:15 brd ff:ff:ff:ff:ff:ff
inet 10.42.8.54/21 brd 10.42.15.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::230:48ff:fe92:f315/64 scope link
valid_lft forever preferred_lft forever
(In reply to comment #17) > (In reply to comment #14) > > Or rename /usr/lib/udev/rules.d/80-net-name-slot.rules to > > /usr/lib/udev/rules.d/76-net-name-slot.rules and of course reboot > > and check if the network has been renamed. > > Yes, better. > > Don't forget to call "mkinitrd" to update the rules in the initrd > (does mkinitrd [dracul?] copy the rules to the initrd?). I have just unpacked the latest initrd, there is no "*-net-name-slot.rules" file. taggart7:/tmp/gg # find . -name \*rules ./etc/udev/rules.d/70-persistent-net.rules ./usr/lib/udev/rules.d/10-dm.rules ./usr/lib/udev/rules.d/60-persistent-storage.rules ./usr/lib/udev/rules.d/50-udev-default.rules ./usr/lib/udev/rules.d/77-network.rules ./usr/lib/udev/rules.d/13-dm-disk.rules ./usr/lib/udev/rules.d/50-firmware.rules ./usr/lib/udev/rules.d/60-persistent-input.rules ./usr/lib/udev/rules.d/61-msft.rules ./usr/lib/udev/rules.d/95-dm-notify.rules ./usr/lib/udev/rules.d/80-drivers.rules Ahh, I get it. You are using eth1 to boot the system?! If so, then most probably that's the issue to rename it after. Per, working on a small patch now for mkinitrd. Will let you know how it comes out :) Per, can you download and try mkinitrd from: https://build.opensuse.org/package/binaries/home:rmilasan:branches:Base:System/mkinitrd?repository=openSUSE_Factory Per, from comment#22, can you check whats in /etc/udev/rules.d/70-persistent-net.rules which you have found in your initrd? *** Bug 834255 has been marked as a duplicate of this bug. *** Robert, as requested, I've installed from the 13.1b1 iso instead, same problem. Created attachment 561133 [details]
60-persistent-input.rules from the initrd
(In reply to comment #29) > Created an attachment (id=561133) [details] > 60-persistent-input.rules from the initrd Ooopss, I guess I need another coffee, obviously a bit too early. Hmm, there is no "/etc/udev/rules.d/70-persistent-net.rules" in the initrd. I don't honestly understand how you can have eth1 (or maybe I do) ?! Is it possible the ISO does not include some rules?! (In reply to comment #31) > I don't honestly understand how you can have eth1 (or maybe I do) ?! > What's so special about eth1? I don't understand your confusion. You shouldn't have it, it should be something like enp14s0. Can you boot the installer in text mode and switch to the console where you can see what network adapters you have (ifconfig -a)? I'm not sure if in graphical mode you can switch to the console. (In reply to comment #33) > You shouldn't have it, it should be something like enp14s0. Haha, yeah, that's what the report is all about :-) As far as I've understood, the problem is that systemd should be renaming both eth0 and eth1 appropriately, but as this doesn't happen in the initrd, only the unused one (eth0) is renamed (later). > Can you boot the installer in text mode and switch to the console where you can > see what network adapters you have (ifconfig -a)? > > I'm not sure if in graphical mode you can switch to the console. I'm always in text-mode anyway, no probs. It'll have to wait until tomorrow, I think. (which time-zone are you on?) (In reply to comment #33) > You shouldn't have it, it should be something like enp14s0. Can you boot the > installer in text mode and switch to the console where you can see what network > adapters you have (ifconfig -a)? > > I'm not sure if in graphical mode you can switch to the console. You can (Ctrl-Alt-Fx), but this is the output of "ifconfig -a" : enp13s0 Link encap:Ethernet HWaddr 00:30:48:92:F3:14 inet6 addr: fe80::230:48ff:fe92:f314/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:256 (256.0 b) TX bytes:2064 (2.0 Kb) Interrupt:11 Memory:ee100000-ee120000 enp14s0 Link encap:Ethernet HWaddr 00:30:48:92:F3:15 inet addr:10.42.8.54 Bcast:10.42.15.255 Mask:255.255.248.0 inet6 addr: fe80::230:48ff:fe92:f315/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:76478 errors:0 dropped:0 overruns:0 frame:0 TX packets:6890 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:115771273 (110.4 Mb) TX bytes:492899 (481.3 Kb) Interrupt:10 Memory:ee200000-ee220000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) eth1 was correctly renamed because the network isn't in use at this point. Now that is strange. You have the network setup correctly, but for whatever reason after the installation enp14s0 ends-up eth1. I will try to reproduce this, is there a specific way you boot and/or setup your machine? (In reply to comment #36) > Now that is strange. You have the network setup correctly, but for whatever > reason after the installation enp14s0 ends-up eth1. Um, that's not really so strange. The installation system is booted over PXE, there is no iSCSI involved at this point. The kernel and the initrd are transferred, then booted - no network active. The driver enumerates the interface first as eth0 and eth1, and they're then appropriately renamed by systemd/udev/whoever. Once the system is installed, it boots over PXE+iSCSI, i.e. first we connect the iSCSI drive as drive 0x80, then the system is booted from that. Because eth1 is now active, it cannot be renamed. > I will try to reproduce this, is there a specific way you boot and/or setup > your machine? PXE+tftp+ipxe+iscsi OK, I've successfully reproduce this bug, you don't need PXE, tftp, ipxe, only iscsi. Seems that the network adapter which is used for iscsi comes up faster then it should or too faster so it would be renamed. It seems that iscsiadm -m node -L onboot is ran before udev, making the rest of the things bad, if you get my drift. I'm adding the maintainer of iscsi and mkinitrd. Will also attach the fix for udev in the initrd to support the new rules for networking. Created attachment 561443 [details]
mkinitrd-support-predictable-names-at-boot.patch
This is the patch to support predictable names in the initrd image.
Lee, can you please take a look at the problem, looks to me like iscsi issue. *** Bug 843665 has been marked as a duplicate of this bug. *** (In reply to comment #41) > Lee, can you please take a look at the problem, looks to me like iscsi issue. Robert: not clear what you are asking. You supplied a patch to initrd udev setup script. Does that fix the problem? I have not been able to reproduce this bug, so I cannot test the patch at this time. Per, please test the provided mkinitrd package. I cant test it myself because interfaces are not renamed in hyperv guests, and due to bnc#846193. (In reply to comment #44) > Per, please test the provided mkinitrd package. I cant test it myself because > interfaces are not renamed in hyperv guests, and due to bnc#846193. Okay - on my system "taggart7", I updated mkinitrd to 2.8 and applied Robert's patch from comment#40, and rebuilt the initrd. The network feature and my e1000e module were not included by default, making the system non bootable. 2nd try - I ran mkinitrd with "-m e1000e -f network", but the system still didn't boot. The network interfaces were renamed, but not brought up. The patch only suppose to help the running system, but if this happens (setup of the network) before udev has the opportunity to rename the network, then must be in PXE initrd or something like that. I really don't know. (In reply to comment #46) > The patch only suppose to help the running system, but if this happens (setup > of the network) before udev has the opportunity to rename the network, then > must be in PXE initrd or something like that. I really don't know. Well, the renaming seems to work, but the interfaces aren't brought up. This worked previously, so I don't understand why it has changed now. Is it possible that the necessary network interface isn't being started due to having been renamed (to enpxxxxx) ? It worked when it was called "eth1". There is no earlier initrd. The system first boots PXE, which chainloads iPXE which in turn hooks up the iSCSI disk and lets the system boot from that. The initrd is loaded from the root filesystem. (In reply to comment #47) > (In reply to comment #46) > > The patch only suppose to help the running system, but if this happens (setup > > of the network) before udev has the opportunity to rename the network, then > > must be in PXE initrd or something like that. I really don't know. > > Well, the renaming seems to work, but the interfaces aren't brought up. This > worked previously, so I don't understand why it has changed now. Is it > possible that the necessary network interface isn't being started due to having > been renamed (to enpxxxxx) ? It worked when it was called "eth1". Bringing up the network is not straight forward if the initrd was built on a system with different network settings, like building it on the nfsroot server. Try to boot with 'interface=eth0 dhcp_macaddresses=eth0' (change eth0 to the actual interface name). If that works, rerun mkinitrd in that state and see if this state gets included into the initrd. I think it should. If there is still an issue what -f network -m e1000 is required, please open a new bug if there isnt one already. Appearently the mkinitrd part of iscsi is not working properly. (In reply to comment #48) > Bringing up the network is not straight forward if the initrd was built on a > system with different network settings, like building it on the nfsroot server. Yes, I understand that, but here I built it on the same system. I kept the working initrd as a backup so I would still have a working system if the new initrd was bad :-) > Try to boot with 'interface=eth0 dhcp_macaddresses=eth0' (change eth0 to the > actual interface name). If that works, rerun mkinitrd in that state and see if > this state gets included into the initrd. I think it should. Hmm, I see - because the working system uses eth1, the initrd would be built to use that too? I tried booting with just "interface=enp14s0", but that made no difference. Then I tried "interface=enp14s0 dhcp_macaddresses=enp14s0", which worked! When I rebuilt the initrd after booting with "interface=enp14s0 dhcp_macaddresses=enp14s0", the network interface module "e1000e" and feature "network" were automatically enabled. The new initrd also works fine, I get the two interfaces with the new names, as expected. @Michal, can you please check the patch from comment #40 and maybe check it into mkinitrd git? Please double check if the fixed mkinitrd.rpm still works (2.8.1 or later): https://build.opensuse.org/package/binaries/Base:System/mkinitrd?repository=openSUSE_Factory Package retrieved from: http://download.opensuse.org/repositories/Base:/System/openSUSE_Factory/i586/mkinitrd-2.8.1-258.1.i586.rpm taggart7:/tmp # rpm --upgrade mkinitrd-2.8.1-258.1.i586.rpm warning: mkinitrd-2.8.1-258.1.i586.rpm: Header V3 RSA/SHA1 Signature, key ID e2c0098c: NOKEY Updating /etc/sysconfig/kernel... Scanning scripts ... Resolve dependencies ... Install symlinks in /lib/mkinitrd/setup ... Install symlinks in /lib/mkinitrd/boot ... Kernel image: /boot/vmlinuz-3.11.1-1.g1383321-pae Initrd image: /boot/initrd-3.11.1-1.g1383321-pae KMS drivers: xgifb Root device: /dev/disk/by-id/scsi-1494554000000000074616767617274370000000000000000-part1 (/dev/sda1) (mounted on / as ext4) Kernel Modules: thermal_sys thermal processor fan ata_piix ata_generic scsi_dh scsi_dh_alua scsi_dh_emc scsi_dh_hp_sw scsi_dh_rdac xgifb scsi_transport_iscsi libiscsi libiscsi_tcp iscsi_tcp usb-common usbcore ohci-hcd uhci-hcd ehci-hcd xhci-hcd usbhid hid-logitech-dj hid-generic hid-holtek-kbd hid-lenovo-tpkbd hid-ortek hid-roccat hid-roccat-common hid-roccat-arvo hid-roccat-isku hid-samsung ehci-pci ohci-pci af_packet pps_core ptp e1000e crc32c-intel uio cnic bnx2i cp: cannot stat ‘/sbin/iscsiuio’: No such file or directory Features: acpi kms block usb network iscsi Looks good. I booted from it, also works. This is an autogenerated message for OBS integration: This bug (820407) was mentioned in https://build.opensuse.org/request/show/203579 Factory / mkinitrd Fixed. |
User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:11.0) Gecko/20100101 Firefox/11.0 Hardware: taggart2, 1U zattoo-box, quad core, 2Gb RAM, dual NICs. Software openSUSE 13.1 M1 Process: network install over pxe+ssh, root on iSCSI. # ip addr 2: enp13s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:30:48:90:80:76 brd ff:ff:ff:ff:ff:ff inet6 fe80::230:48ff:fe90:8076/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:30:48:90:80:77 brd ff:ff:ff:ff:ff:ff inet 10.42.8.94/21 brd 10.42.15.255 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::230:48ff:fe90:8077/64 scope link valid_lft forever preferred_lft forever # l /etc/sysconfig/network/ total 100 drwxr-xr-x 6 root root 4096 May 17 12:50 ./ drwxr-xr-x 5 root root 4096 May 17 12:41 ../ -rw-r--r-- 1 root root 12911 May 17 12:50 config -rw-r--r-- 1 root root 10592 May 17 12:50 dhcp -rw-r--r-- 1 root root 193 May 17 12:50 ifcfg-enp13s0 -rw-r--r-- 1 root root 80 May 17 12:40 ifcfg-enp14s0 -rw------- 1 root root 151 Mar 6 02:57 ifcfg-lo -rw-r--r-- 1 root root 29387 Mar 6 02:57 ifcfg.template drwxr-xr-x 2 root root 4096 May 17 12:38 if-down.d/ -rw-r--r-- 1 root root 239 Mar 6 02:57 ifroute-lo drwxr-xr-x 2 root root 4096 May 17 12:39 if-up.d/ drwx------ 2 root root 4096 Feb 3 08:16 providers/ drwxr-xr-x 2 root root 4096 May 17 12:39 scripts/ Where did that 'eth1' come from?? Also, for some reason both interfaces are configured with BOOTPROTO='dhcp' and STARTMODE='nfsroot', although I only configured one during installation. Reproducible: Always