|
Bugzilla – Full Text Bug Listing |
| Summary: | No network (wicked) after 23-12-2022 updates - bond change in kernel | ||
|---|---|---|---|
| Product: | [openSUSE] openSUSE Tumbleweed | Reporter: | Alessandro Sturniolo <alessandro.sturniolo> |
| Component: | Network | Assignee: | wicked maintainers <wicked-maintainers> |
| Status: | RESOLVED FIXED | QA Contact: | E-mail List <qa-bugs> |
| Severity: | Major | ||
| Priority: | P5 - None | CC: | alex.bobrov, arvidjaar, email, kernel-bugs, mt, simon, wmicha |
| Version: | Current | Flags: | mt:
needinfo?
(kernel-bugs) |
| Target Milestone: | --- | ||
| Hardware: | x86-64 | ||
| OS: | openSUSE Tumbleweed | ||
| Whiteboard: | |||
| Found By: | --- | Services Priority: | |
| Business Priority: | Blocker: | --- | |
| Marketing QA Status: | --- | IT Deployment: | --- |
| Attachments: |
Network related log files and configurations.
Test script: bond-ip.sh [-1 ethX -2 ethY] [-m mode] [one|two|] Test script outputs with 4.x, 6.0 and 6.1 cases |
||
|
Description
Alessandro Sturniolo
2022-12-23 12:53:39 UTC
Similarly
# journalctl | grep bond0
дек 28 12:04:41 srv-mail wicked[1801]: received new ready device bond0 (/org/opensuse/Network/Interface/5)
дек 28 12:04:41 srv-mail wicked[1801]: bond0: refresh <client-state> control: persistent=false, usercontrol=false, require-link=auto
дек 28 12:04:41 srv-mail wicked[1801]: bond0: refresh <client-state> config: origin=compat:suse:/etc/sysconfig/network/ifcfg-bond0, uuid=e483643d-7a31-5cd2-8932-39fd7ab4793b, owner-uid=4294967295
дек 28 12:04:41 srv-mail wicked[1801]: bond0: changed state none -> device-exists
дек 28 12:04:41 srv-mail wicked[1801]: ni_sysconfig_read(/etc/sysconfig/network/ifcfg-bond0)
дек 28 12:04:41 srv-mail wicked[1801]: ni_compat_generate_interfaces: compat:suse:/etc/sysconfig/network/ifcfg-bond0:0
дек 28 12:04:41 srv-mail wicked[1801]: bond0: <device> identified device as "enp15s0f1"
дек 28 12:04:41 srv-mail wicked[1801]: bond0: resolved reference bond/slaves/slave/device to subordinate device enp15s0f1
дек 28 12:04:41 srv-mail wicked[1801]: enp15s0f1 (<orphan xml node>): setting master device to bond0
дек 28 12:04:41 srv-mail wicked[1801]: bond0 [5]
дек 28 12:04:41 srv-mail wicked[1801]: bond0: hiring nanny
дек 28 12:04:41 srv-mail wicked[1801]: bond0: converting config into policy 'policy__bond0'
дек 28 12:04:41 srv-mail wicked[1801]: bond0: adding policy policy__bond0 to nanny
дек 28 12:04:41 srv-mail wicked[1801]: Calling /org/opensuse/Network/Nanny.createPolicy(policy__bond0)
дек 28 12:04:41 srv-mail wicked[1801]: Call to /org/opensuse/Network/Nanny.createPolicy(policy__bond0) failed: Policy already exists
дек 28 12:04:41 srv-mail wicked[1801]: Calling /org/opensuse/Network/Nanny/Policy/policy__bond0.update()
дек 28 12:04:41 srv-mail wicked[1801]: Successfully updated nanny policy /org/opensuse/Network/Nanny/Policy/policy__bond0
дек 28 12:04:41 srv-mail wicked[1801]: bond0: nanny hired!
дек 28 12:04:41 srv-mail wicked[1801]: bond0: configuration applied to nanny
дек 28 12:04:48 srv-mail wicked[1801]: received refresh for ready device bond0 (/org/opensuse/Network/Interface/5)
дек 28 12:04:48 srv-mail wicked[1801]: bond0: refresh <client-state> control: persistent=false, usercontrol=false, require-link=auto
дек 28 12:04:48 srv-mail wicked[1801]: bond0: refresh <client-state> config: origin=compat:suse:/etc/sysconfig/network/ifcfg-bond0, uuid=e483643d-7a31-5cd2-8932-39fd7ab4793b, owner-uid=4294967295
дек 28 12:04:48 srv-mail wicked[1801]: received refresh for ready device bond0 (/org/opensuse/Network/Interface/5)
дек 28 12:04:48 srv-mail wicked[1801]: bond0: refresh <client-state> control: persistent=false, usercontrol=false, require-link=auto
дек 28 12:04:48 srv-mail wicked[1801]: bond0: refresh <client-state> config: origin=compat:suse:/etc/sysconfig/network/ifcfg-bond0, uuid=e483643d-7a31-5cd2-8932-39fd7ab4793b, owner-uid=4294967295
# ip a
3: enp15s0f1: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc pfifo_fast master bond0 state DOWN group default qlen 1000
link/ether 00:e0:81:b1:c0:1d brd ff:ff:ff:ff:ff:ff
............
5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 7a:fa:1d:9a:b8:0f brd ff:ff:ff:ff:ff:ff
Same here, is reproduceable if you have a "bond" configuration (normal hardware configuration will work) (according to andrei at mailinglist users@lists.opensuse.org) long (about one year) not updated tumbleweed was running fine. then update to tumbleweed 20221228 and network is not working. now at 20230102 still same. here what i have, and a workaround later in this comment: if it is working, it looks like this: ================================================ wicked ifstatus all lo up link: #1, state up type: loopback config: compat:suse:/etc/sysconfig/network/ifcfg-lo leases: ipv4 static granted leases: ipv6 static granted addr: ipv4 127.0.0.1/8 [static] addr: ipv6 ::1/128 [static] enp3s0 enslaved link: #2, state up, mtu 1500, master bond0 type: ethernet, hwaddr 18:67:b0:b9:a6:58 config: compat:suse:/etc/sysconfig/network/ifcfg-enp3s0 wlp2s0 up link: #3, state up, mtu 1500 type: wireless, state established, ssid XAuaLiwaru9835, WPA2-PSK+WPA-PSK config: compat:suse:/etc/sysconfig/network/ifcfg-wlp2s0 bond0 up link: #4, state up, mtu 1500 type: bond, mode active-backup, hwaddr 18:67:b0:b9:a6:58 config: compat:suse:/etc/sysconfig/network/ifcfg-bond0 leases: ipv4 static granted addr: ipv4 192.168.0.23/24 [static] route: ipv4 default via 192.168.0.99 [static] ==================================== it is configured that if cable (enp3s0) is plugged in, it will use cable, and if not plugged in it will switch to whireless (wlp2s0) ==================================== if its not working (before starting manually) it has the status: "device-not-running" for enp3s0 wlp2s0 and bond0 ==================================== after playing around i found out that the command: ip link set enp3s0 up ip link set wlp2s0 up will bring the network cards UP, but this i have to enter manually, AND wicked ifup enp3s0 wicked ifup wlp2s0 will NOT bring the network cards UP. (therefore wicked will during boot hang and run out of time, then boot continues ========================================== here from boot: Dez 29 21:54:08 becherer7 systemd[1]: Starting wicked AutoIPv4 supplicant service... Dez 29 21:54:08 becherer7 systemd[1]: Starting wicked DHCPv4 supplicant service... Dez 29 21:54:08 becherer7 systemd[1]: Starting wicked DHCPv6 supplicant service... Dez 29 21:54:08 becherer7 systemd[1]: Started wicked DHCPv6 supplicant service. Dez 29 21:54:08 becherer7 systemd[1]: Started wicked DHCPv4 supplicant service. Dez 29 21:54:08 becherer7 systemd[1]: Started wicked AutoIPv4 supplicant service. Dez 29 21:54:08 becherer7 systemd[1]: Starting wicked network management service daemon... Dez 29 21:54:08 becherer7 systemd[1]: Started wicked network management service daemon. Dez 29 21:54:08 becherer7 systemd[1]: Starting wicked network nanny service... Dez 29 21:54:08 becherer7 systemd[1]: Started wicked network nanny service. Dez 29 21:54:08 becherer7 systemd[1]: Starting wicked managed network interfaces... Dez 29 21:54:38 becherer7 wicked[1227]: lo up Dez 29 21:54:38 becherer7 wicked[1227]: enp3s0 device-not-running Dez 29 21:54:38 becherer7 wicked[1227]: wlp2s0 device-not-running Dez 29 21:54:38 becherer7 wicked[1227]: bond0 device-not-running Dez 29 21:54:38 becherer7 systemd[1]: Finished wicked managed network interfaces. ================================================= and here when i try manually using wicked: Dez 29 22:19:14 becherer7 wickedd-nanny[1226]: device wlp2s0: call to org.opensuse.Network.Interface.waitLinkUp() failed: General failure Dez 29 22:19:14 becherer7 wickedd-nanny[1226]: wlp2s0: failed to bring up device, still continuing Dez 29 22:19:19 becherer7 wickedd-nanny[1226]: device enp3s0: call to org.opensuse.Network.Interface.waitLinkUp() failed: General failure Dez 29 22:19:19 becherer7 wickedd-nanny[1226]: enp3s0: failed to bring up device, still continuing =================================================== workaround for boot: as allesandro sturniolo wrote, use ip link set enp3s0 up ip link set wlp2s0 up (in my setup) so make a file netz-wicked-workaround.service --------file ---------------------------- [Unit] Description=netz-wicked-workaround After=wicked.service [Service] Type=simple ExecStart=/usr/bin/netz RemainAfterExit=no [Install] WantedBy=multi-user.target -------endfile -------------------------------- copy this to /etc/systemd/system -------------------------------------- make a file netz -------------- file (for my setup)------------- #!/bin/bash sudo ip link set wlp2s0 up sudo ip link set enp3s0 up ----------- endfile-------------------- copy it to /usr/bin/netz and give them executable rights ---------------------------------------- use yast2 to start the new service at boot netz-wicked-workaround.service ------------------------------------- simoN (In reply to Alessandro Sturniolo from comment #0) > > Today (23-12-2022) I have updated my system, and after reboot (there was a > new kernel version), network no longer worked. > You updated from what snapshot exactly? What was the previous version of wicked? What packages were updated? I suspect the problem is not wicked (at least, directly) but missing event or notification. So the reason could be kernel or udev/systemd. It would be extremely helpful if you could identify what package exactly caused this regression. Get the list of packages that were updated; revert to previous snapshot; update each package individually and check whether you observe this problem. (In reply to Andrei Borzenkov from comment #3) > (In reply to Alessandro Sturniolo from comment #0) > > > > Today (23-12-2022) I have updated my system, and after reboot (there was a > > new kernel version), network no longer worked. > > > > You updated from what snapshot exactly? What was the previous version of > wicked? What packages were updated? I don't know. I do a zypper dup, almost once a week. From the logs, on 23-12-2022 I've updated following packages: CoreFreq-1.93.1-41.5.x86_64 CoreFreq-1.93.1-41.6.x86_64 CoreFreq-kmp-default-1.92.4_k6.0.10_1-39.2.x86_64 CoreFreq-kmp-default-1.93.1_k6.0.12_1-41.5.x86_64 CoreFreq-kmp-default-1.93.1_k6.0.12_1-41.6.x86_64 MozillaFirefox-107.0.1-1.1.x86_64 MozillaFirefox-108.0.1-1.1.x86_64 avahi-0.8-21.1.x86_64 avahi-0.8-22.1.x86_64 cairomm1_0-devel-1.12.2-3.22.x86_64 cairomm1_0-devel-1.12.2-4.1.x86_64 chromium-108.0.5359.124-1.1.x86_64 chromium-108.0.5359.124-2.1.x86_64 colord-1.4.6-204.13.x86_64 colord-1.4.6-204.14.x86_64 colord-color-profiles-1.4.6-204.13.x86_64 colord-color-profiles-1.4.6-204.14.x86_64 cpp13-13.0.0+git197351-1.1.x86_64 cpp13-13.0.0+git197351-2.1.x86_64 dxvk-1.10.2-74.243.x86_64 dxvk-1.10.2-74.244.x86_64 dxvk-32bit-1.10.2-74.243.x86_64 dxvk-32bit-1.10.2-74.245.x86_64 enchant-1-backends-1.6.1-5.15.x86_64 enchant-1-backends-1.6.1-6.1.x86_64 ethtool-6.0-1.1.x86_64 ethtool-6.1-1.1.x86_64 gcc13-13.0.0+git197351-1.1.x86_64 gcc13-13.0.0+git197351-2.1.x86_64 gcc13-32bit-13.0.0+git197351-1.1.x86_64 gcc13-32bit-13.0.0+git197351-2.1.x86_64 gcc13-fortran-13.0.0+git197351-1.1.x86_64 gcc13-fortran-13.0.0+git197351-2.1.x86_64 gcc13-fortran-32bit-13.0.0+git197351-1.1.x86_64 gcc13-fortran-32bit-13.0.0+git197351-2.1.x86_64 gcc13-locale-13.0.0+git197351-1.1.x86_64 gcc13-locale-13.0.0+git197351-2.1.x86_64 gdk-pixbuf-loader-libopenraw-0.3.1-54.105.x86_64 gdk-pixbuf-loader-libopenraw-0.3.1-54.106.x86_64 gio-branding-openSUSE-42.1-18.1.noarch gio-branding-openSUSE-42.1-19.1.noarch glibc-2.36-8.1.x86_64 glibc-2.36-9.1.x86_64 glibc-32bit-2.36-8.1.x86_64 glibc-32bit-2.36-9.1.x86_64 glibc-devel-2.36-8.1.x86_64 glibc-devel-2.36-9.1.x86_64 glibc-devel-32bit-2.36-8.1.x86_64 glibc-devel-32bit-2.36-9.1.x86_64 glibc-extra-2.36-8.1.x86_64 glibc-extra-2.36-9.1.x86_64 glibc-i18ndata-2.36-8.1.noarch glibc-i18ndata-2.36-9.1.noarch glibc-info-2.36-8.1.noarch glibc-info-2.36-9.1.noarch glibc-locale-2.36-8.1.x86_64 glibc-locale-2.36-9.1.x86_64 glibc-locale-base-2.36-8.1.x86_64 glibc-locale-base-2.36-9.1.x86_64 glibc-locale-base-32bit-2.36-8.1.x86_64 glibc-locale-base-32bit-2.36-9.1.x86_64 glibc-profile-2.36-8.1.x86_64 glibc-profile-2.36-9.1.x86_64 glibc-profile-32bit-2.36-8.1.x86_64 glibc-profile-32bit-2.36-9.1.x86_64 imlib2-1.10.0-1.1.x86_64 imlib2-1.9.1-2.1.x86_64 imlib2-filters-1.10.0-1.1.x86_64 imlib2-filters-1.9.1-2.1.x86_64 imlib2-loaders-1.10.0-1.1.x86_64 imlib2-loaders-1.9.1-2.1.x86_64 kbd-2.4.0-10.2.x86_64 kbd-2.4.0-10.3.x86_64 kbd-legacy-2.4.0-10.2.noarch kbd-legacy-2.4.0-10.3.noarch kernel-default-6.0.10-1.1.x86_64 kernel-default-6.1.0-1.1.x86_64 kernel-default-devel-6.0.10-1.1.x86_64 kernel-default-devel-6.1.0-1.1.x86_64 kernel-devel-6.0.10-1.1.noarch kernel-devel-6.1.0-1.1.noarch kernel-docs-6.0.12-1.1.noarch kernel-docs-6.1.0-1.1.noarch kernel-macros-6.0.12-1.1.noarch kernel-macros-6.1.0-1.1.noarch kernel-source-6.0.10-1.1.noarch kernel-source-6.1.0-1.1.noarch kernel-syms-6.0.10-1.1.x86_64 kernel-syms-6.1.0-1.1.x86_64 libImlib2-1-1.10.0-1.1.x86_64 libImlib2-1-1.9.1-2.1.x86_64 libart_lgpl-devel-2.3.21-24.7.x86_64 libart_lgpl-devel-2.3.21-25.1.x86_64 libart_lgpl_2-2-2.3.21-24.7.x86_64 libart_lgpl_2-2-2.3.21-25.1.x86_64 libasan8-13.0.0+git197351-1.1.x86_64 libasan8-13.0.0+git197351-2.1.x86_64 libasan8-32bit-13.0.0+git197351-1.1.x86_64 libasan8-32bit-13.0.0+git197351-2.1.x86_64 libatomic1-13.0.0+git197351-1.1.x86_64 libatomic1-13.0.0+git197351-2.1.x86_64 libatomic1-32bit-13.0.0+git197351-1.1.x86_64 libatomic1-32bit-13.0.0+git197351-2.1.x86_64 libavahi-client3-0.8-21.1.x86_64 libavahi-client3-0.8-22.1.x86_64 libavahi-client3-32bit-0.8-21.1.x86_64 libavahi-client3-32bit-0.8-22.1.x86_64 libavahi-common3-0.8-21.1.x86_64 libavahi-common3-0.8-22.1.x86_64 libavahi-common3-32bit-0.8-21.1.x86_64 libavahi-common3-32bit-0.8-22.1.x86_64 libavahi-core7-0.8-21.1.x86_64 libavahi-core7-0.8-22.1.x86_64 libavahi-devel-0.8-21.1.x86_64 libavahi-devel-0.8-22.1.x86_64 libavahi-glib1-0.8-21.1.x86_64 libavahi-glib1-0.8-22.1.x86_64 libavahi-libevent1-0.8-21.1.x86_64 libavahi-libevent1-0.8-22.1.x86_64 libcairomm-1_0-1-1.12.2-3.22.x86_64 libcairomm-1_0-1-1.12.2-4.1.x86_64 libcloudproviders0-0.3.1-2.11.x86_64 libcloudproviders0-0.3.1-3.1.x86_64 libcolorhug2-1.4.6-204.13.x86_64 libcolorhug2-1.4.6-204.14.x86_64 libdns_sd-0.8-21.1.x86_64 libdns_sd-0.8-22.1.x86_64 libenchant1-1.6.1-5.15.x86_64 libenchant1-1.6.1-6.1.x86_64 libgcc_s1-13.0.0+git197351-1.1.x86_64 libgcc_s1-13.0.0+git197351-2.1.x86_64 libgcc_s1-32bit-13.0.0+git197351-1.1.x86_64 libgcc_s1-32bit-13.0.0+git197351-2.1.x86_64 libgfortran5-13.0.0+git197351-1.1.x86_64 libgfortran5-13.0.0+git197351-2.1.x86_64 libgfortran5-32bit-13.0.0+git197351-1.1.x86_64 libgfortran5-32bit-13.0.0+git197351-2.1.x86_64 libglade-2_0-0-2.6.4-27.9.x86_64 libglade-2_0-0-2.6.4-28.1.x86_64 libgmic3-3.1.6-28.10.x86_64 libgmic3-3.1.6-29.1.x86_64 libgomp1-13.0.0+git197351-1.1.x86_64 libgomp1-13.0.0+git197351-2.1.x86_64 libgomp1-32bit-13.0.0+git197351-1.1.x86_64 libgomp1-32bit-13.0.0+git197351-2.1.x86_64 libgtop-2_0-11-2.40.0-3.6.x86_64 libgtop-2_0-11-2.40.0-4.1.x86_64 libgtop-lang-2.40.0-3.6.noarch libgtop-lang-2.40.0-4.1.noarch libhdf5-200-1.12.2-199.17.x86_64 libhdf5-200-1.12.2-199.19.x86_64 libhdf5_cpp200-1.12.2-199.17.x86_64 libhdf5_cpp200-1.12.2-199.19.x86_64 libhdf5_fortran200-1.12.2-199.17.x86_64 libhdf5_fortran200-1.12.2-199.19.x86_64 libhdf5_hl200-1.12.2-199.17.x86_64 libhdf5_hl200-1.12.2-199.19.x86_64 libhdf5_hl_cpp200-1.12.2-199.17.x86_64 libhdf5_hl_cpp200-1.12.2-199.19.x86_64 libhdf5hl_fortran200-1.12.2-199.17.x86_64 libhdf5hl_fortran200-1.12.2-199.19.x86_64 libhwasan0-13.0.0+git197351-1.1.x86_64 libhwasan0-13.0.0+git197351-2.1.x86_64 libitm1-13.0.0+git197351-1.1.x86_64 libitm1-13.0.0+git197351-2.1.x86_64 libitm1-32bit-13.0.0+git197351-1.1.x86_64 libitm1-32bit-13.0.0+git197351-2.1.x86_64 liblsan0-13.0.0+git197351-1.1.x86_64 liblsan0-13.0.0+git197351-2.1.x86_64 liblzma5-32bit-5.2.10-1.1.x86_64 liblzma5-32bit-5.2.8-2.1.x86_64 liblzma5-5.2.10-1.1.x86_64 liblzma5-5.2.8-2.1.x86_64 libmbedcrypto7-2.28.1-1.1.x86_64 libmbedcrypto7-2.28.2-1.1.x86_64 libmbedtls14-2.28.1-1.1.x86_64 libmbedtls14-2.28.2-1.1.x86_64 libmbedx509-1-2.28.1-1.1.x86_64 libmbedx509-1-2.28.2-1.1.x86_64 libmpc3-1.3.0-1.1.x86_64 libmpc3-1.3.1-1.1.x86_64 libnm0-1.40.6-1.1.x86_64 libnm0-1.40.8-1.1.x86_64 libnm0-32bit-1.40.6-1.1.x86_64 libnm0-32bit-1.40.8-1.1.x86_64 libobjc4-13.0.0+git197351-1.1.x86_64 libobjc4-13.0.0+git197351-2.1.x86_64 libosip2-15-5.3.0-1.4.x86_64 libosip2-15-5.3.1-1.1.x86_64 libosip2-devel-5.3.0-1.4.x86_64 libosip2-devel-5.3.1-1.1.x86_64 libpangomm-1_4-1-2.46.3-1.1.x86_64 libpangomm-1_4-1-2.46.3-2.1.x86_64 libpcre2-16-0-10.42-2.1.x86_64 libpcre2-16-0-10.42-3.1.x86_64 libpcre2-16-0-32bit-10.42-2.1.x86_64 libpcre2-16-0-32bit-10.42-3.1.x86_64 libpcre2-32-0-10.42-2.1.x86_64 libpcre2-32-0-10.42-3.1.x86_64 libpcre2-8-0-10.42-2.1.x86_64 libpcre2-8-0-10.42-3.1.x86_64 libpcre2-8-0-32bit-10.42-71.1.x86_64 libpcre2-8-0-32bit-10.42-73.1.x86_64 libpcre2-posix3-10.42-2.1.x86_64 libpcre2-posix3-10.42-3.1.x86_64 libquadmath0-13.0.0+git197351-1.1.x86_64 libquadmath0-13.0.0+git197351-2.1.x86_64 libquadmath0-32bit-13.0.0+git197351-1.1.x86_64 libquadmath0-32bit-13.0.0+git197351-2.1.x86_64 librpmbuild9-4.18.0-1.1.x86_64 librpmbuild9-4.18.0-2.1.x86_64 libtsan2-13.0.0+git197351-1.1.x86_64 libtsan2-13.0.0+git197351-2.1.x86_64 libubsan1-13.0.0+git197351-1.1.x86_64 libubsan1-13.0.0+git197351-2.1.x86_64 libubsan1-32bit-13.0.0+git197351-1.1.x86_64 libubsan1-32bit-13.0.0+git197351-2.1.x86_64 libvlc5-3.0.18-6.2.x86_64 libvlc5-3.0.18-7.1.x86_64 libvlccore9-3.0.18-6.2.x86_64 libvlccore9-3.0.18-7.1.x86_64 lutris-0.5.12-77.7.noarch lutris-0.5.12-77.8.noarch lutris-0.5.12-77.9.noarch nodejs19-19.3.0-20.1.x86_64 nodejs19-19.3.0-20.2.x86_64 npm19-19.3.0-20.1.x86_64 npm19-19.3.0-20.2.x86_64 nscd-2.36-8.1.x86_64 nscd-2.36-9.1.x86_64 openSUSE-release-20221220-2007.1.x86_64 openSUSE-release-20221222-2010.1.x86_64 openSUSE-release-ftp-20221220-2007.1.x86_64 openSUSE-release-ftp-20221222-2010.1.x86_64 pangomm1_4-devel-2.46.3-1.1.x86_64 pangomm1_4-devel-2.46.3-2.1.x86_64 patch-2.7.6-72.d_t.182.x86_64 patch-2.7.6-72.d_t.184.x86_64 pcre2-devel-10.42-2.1.x86_64 pcre2-devel-10.42-3.1.x86_64 perf-6.0.12-78.21.x86_64 perf-6.1.0-78.22.x86_64 perf-gtk-6.0.12-78.21.x86_64 perf-gtk-6.1.0-78.22.x86_64 python310-rpm-4.18.0-1.1.x86_64 python310-rpm-4.18.0-2.1.x86_64 python38-rpm-4.18.0-1.1.x86_64 python38-rpm-4.18.0-2.1.x86_64 rpm-32bit-4.18.0-1.1.x86_64 rpm-32bit-4.18.0-2.1.x86_64 rpm-4.18.0-1.1.x86_64 rpm-4.18.0-2.1.x86_64 rpm-build-4.18.0-1.1.x86_64 rpm-build-4.18.0-2.1.x86_64 rpm-build-perl-4.18.0-1.1.x86_64 rpm-build-perl-4.18.0-2.1.x86_64 rpmlint-2.4.0+git20221209.975098b-1.7.noarch rpmlint-2.4.0+git20221209.975098b-1.9.noarch supertuxkart-data-1.4-79.4.noarch supertuxkart-data-1.4-79.5.noarch typelib-1_0-NM-1_0-1.40.6-1.1.x86_64 typelib-1_0-NM-1_0-1.40.8-1.1.x86_64 v4l2loopback-utils-0.12.7-1.12.noarch v4l2loopback-utils-0.12.7-1.13.noarch virtualbox-kmp-default-7.0.4_k6.0.10_1-2.1.x86_64 virtualbox-kmp-default-7.0.4_k6.1.0_1-2.3.x86_64 vlc-3.0.18-6.2.x86_64 vlc-3.0.18-7.1.x86_64 vlc-codec-gstreamer-3.0.18-6.2.x86_64 vlc-codec-gstreamer-3.0.18-7.1.x86_64 vlc-codecs-3.0.18-6.2.x86_64 vlc-codecs-3.0.18-7.1.x86_64 vlc-lang-3.0.18-6.2.noarch vlc-lang-3.0.18-7.1.noarch vlc-noX-3.0.18-6.2.x86_64 vlc-noX-3.0.18-7.1.x86_64 vlc-opencv-3.0.18-6.2.x86_64 vlc-opencv-3.0.18-7.1.x86_64 vlc-qt-3.0.18-6.2.x86_64 vlc-qt-3.0.18-7.1.x86_64 xz-5.2.10-1.1.x86_64 xz-5.2.8-2.1.x86_64 xz-devel-5.2.10-1.1.x86_64 xz-devel-5.2.8-2.1.x86_64 xz-lang-5.2.10-1.1.noarch xz-lang-5.2.8-2.1.noarch and as you can see, wicked is not in the list (and neither udev o systemd). Instead there is a new kernel. > I suspect the problem is not wicked (at least, directly) but missing event > or notification. So the reason could be kernel or udev/systemd. It would be > extremely helpful if you could identify what package exactly caused this > regression. > > Get the list of packages that were updated; revert to previous snapshot; > update each package individually and check whether you observe this problem. I'm not on BTRFS, nor LVM (my system was installed about 11 years ago with ext4... and never formatted or reinstalled), so I think I cannot rollback to a previous snapshot. (In reply to Alessandro Sturniolo from comment #4) > kernel-default-6.0.10-1.1.x86_64 > kernel-default-6.1.0-1.1.x86_64 Yes, I can reproduce it with kernel 6.1.1 and the problem goes away when I boot kernel 6.0.3 without any change in configuration. Apparently something changed between kernel 6.0 and 6.1. Whether it is a bug that must be fixed in kernel or behavior change that must be fixed in wicked I do not know. https://bugzilla.kernel.org/show_bug.cgi?id=216844 Is it a Mediatek MT7921? (In reply to Graeme Spence from comment #6) > Is it a Mediatek MT7921? No. It is trivially reproducible in QEMU with virtio network. Hello. Same here. Kernel 6.1.2 bond with enp4s0f0 and enp4s0f1 fails Kernel 6.0.12 works fine. Kernel 6.1.2: echidna:/var/log # journalctl -b -1 | egrep '(wicked|igb)' Jan 05 07:06:49 echidna kernel: igb: Intel(R) Gigabit Ethernet Network Driver Jan 05 07:06:49 echidna kernel: igb: Copyright (c) 2007-2014 Intel Corporation. Jan 05 07:06:49 echidna kernel: igb 0000:04:00.0: added PHC on eth0 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:06:49 echidna kernel: igb 0000:04:00.0: eth0: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b0 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.0: eth0: PBA No: G15139-002 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:06:49 echidna kernel: igb 0000:04:00.1: added PHC on eth1 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:06:49 echidna kernel: igb 0000:04:00.1: eth1: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b1 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.1: eth1: PBA No: G15139-002 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:06:49 echidna kernel: igb 0000:04:00.2: added PHC on eth2 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.2: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:06:49 echidna kernel: igb 0000:04:00.2: eth2: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b2 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.2: eth2: PBA No: G15139-002 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:06:49 echidna kernel: igb 0000:04:00.3: added PHC on eth4 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.3: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:06:49 echidna kernel: igb 0000:04:00.3: eth4: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b3 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.3: eth4: PBA No: G15139-002 Jan 05 07:06:49 echidna kernel: igb 0000:04:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:06:50 echidna kernel: igb 0000:04:00.0 enp4s0f0: renamed from eth0 Jan 05 07:06:50 echidna kernel: igb 0000:04:00.1 enp4s0f1: renamed from eth1 Jan 05 07:06:50 echidna kernel: igb 0000:04:00.2 enp4s0f2: renamed from eth2 Jan 05 07:06:50 echidna kernel: igb 0000:04:00.3 enp4s0f3: renamed from eth4 Jan 05 07:08:08 echidna systemd[1]: Starting wicked AutoIPv4 supplicant service... Jan 05 07:08:08 echidna systemd[1]: Starting wicked DHCPv4 supplicant service... Jan 05 07:08:08 echidna systemd[1]: Starting wicked DHCPv6 supplicant service... Jan 05 07:08:08 echidna systemd[1]: Started wicked AutoIPv4 supplicant service. Jan 05 07:08:08 echidna systemd[1]: Started wicked DHCPv4 supplicant service. Jan 05 07:08:08 echidna systemd[1]: Started wicked DHCPv6 supplicant service. Jan 05 07:08:08 echidna systemd[1]: Starting wicked network management service daemon... Jan 05 07:08:08 echidna systemd[1]: Started wicked network management service daemon. Jan 05 07:08:08 echidna systemd[1]: Starting wicked network nanny service... Jan 05 07:08:08 echidna systemd[1]: Started wicked network nanny service. Jan 05 07:08:08 echidna systemd[1]: Starting wicked managed network interfaces... Jan 05 07:08:38 echidna wicked[1306]: lo up Jan 05 07:08:38 echidna wicked[1306]: enp4s0f0 device-not-running Jan 05 07:08:38 echidna wicked[1306]: enp4s0f1 device-not-running Jan 05 07:08:38 echidna wicked[1306]: enp4s0f2 setup-in-progress Jan 05 07:08:38 echidna wicked[1306]: enp0s31f6 setup-in-progress Jan 05 07:08:38 echidna wicked[1306]: br0 device-not-running Jan 05 07:08:38 echidna wicked[1306]: bond0 device-not-running Jan 05 07:08:38 echidna systemd[1]: Finished wicked managed network interfaces. Kernel 6.0.12: echidna:/var/log # journalctl -b 0 | egrep '(wicked|igb)' Jan 05 07:15:12 echidna kernel: igb: Intel(R) Gigabit Ethernet Network Driver Jan 05 07:15:12 echidna kernel: igb: Copyright (c) 2007-2014 Intel Corporation. Jan 05 07:15:12 echidna kernel: igb 0000:04:00.0: added PHC on eth0 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:15:12 echidna kernel: igb 0000:04:00.0: eth0: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b0 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.0: eth0: PBA No: G15139-002 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:15:12 echidna kernel: igb 0000:04:00.1: added PHC on eth1 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:15:12 echidna kernel: igb 0000:04:00.1: eth1: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b1 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.1: eth1: PBA No: G15139-002 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:15:12 echidna kernel: igb 0000:04:00.2: added PHC on eth2 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.2: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:15:12 echidna kernel: igb 0000:04:00.2: eth2: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b2 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.2: eth2: PBA No: G15139-002 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:15:12 echidna kernel: igb 0000:04:00.3: added PHC on eth4 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.3: Intel(R) Gigabit Ethernet Network Connection Jan 05 07:15:12 echidna kernel: igb 0000:04:00.3: eth4: (PCIe:5.0Gb/s:Width x4) a0:36:9f:6a:39:b3 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.3: eth4: PBA No: G15139-002 Jan 05 07:15:12 echidna kernel: igb 0000:04:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) Jan 05 07:15:13 echidna kernel: igb 0000:04:00.0 enp4s0f0: renamed from eth0 Jan 05 07:15:13 echidna kernel: igb 0000:04:00.1 enp4s0f1: renamed from eth1 Jan 05 07:15:13 echidna kernel: igb 0000:04:00.3 enp4s0f3: renamed from eth4 Jan 05 07:15:13 echidna kernel: igb 0000:04:00.2 enp4s0f2: renamed from eth2 Jan 05 07:16:31 echidna systemd[1]: Starting wicked AutoIPv4 supplicant service... Jan 05 07:16:31 echidna systemd[1]: Starting wicked DHCPv4 supplicant service... Jan 05 07:16:31 echidna systemd[1]: Starting wicked DHCPv6 supplicant service... Jan 05 07:16:31 echidna systemd[1]: Started wicked DHCPv4 supplicant service. Jan 05 07:16:31 echidna systemd[1]: Started wicked AutoIPv4 supplicant service. Jan 05 07:16:31 echidna systemd[1]: Started wicked DHCPv6 supplicant service. Jan 05 07:16:31 echidna systemd[1]: Starting wicked network management service daemon... Jan 05 07:16:31 echidna systemd[1]: Started wicked network management service daemon. Jan 05 07:16:31 echidna systemd[1]: Starting wicked network nanny service... Jan 05 07:16:31 echidna systemd[1]: Started wicked network nanny service. Jan 05 07:16:31 echidna systemd[1]: Starting wicked managed network interfaces... Jan 05 07:16:37 echidna kernel: igb 0000:04:00.1 enp4s0f1: igb: enp4s0f1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jan 05 07:16:37 echidna kernel: igb 0000:04:00.0 enp4s0f0: igb: enp4s0f0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Jan 05 07:16:37 echidna wickedd-dhcp4[1259]: br0: Request to acquire DHCPv4 lease with UUID 3f6bb663-0d42-0b00-f304-00000e000000 Jan 05 07:16:37 echidna wickedd-dhcp6[1261]: br0: Request to acquire DHCPv6 lease with UUID 3f6bb663-0d42-0b00-f304-00000f000000 in mode auto Jan 05 07:16:49 echidna wickedd-dhcp4[1259]: br0: Committed DHCPv4 lease with address 192.168.200.10 (lease time 863988, renew in 431988 sec, rebind in 755988 sec) Jan 05 07:16:54 echidna wickedd-dhcp6[1261]: br0: Committing DHCPv6 lease with: Jan 05 07:16:54 echidna wickedd-dhcp6[1261]: br0 +ia-na.address 2003:f1:f71f:f100:60d7:f1ff:fed9:ab55/0, pref-lft 3598, valid-lft 7198 Jan 05 07:16:54 echidna wickedd[1267]: br0: address 2003:f1:f71f:f100:60d7:f1ff:fed9:ab55 covered by a auto lease Jan 05 07:17:01 echidna wicked[1279]: lo up Jan 05 07:17:01 echidna wicked[1279]: enp4s0f0 enslaved Jan 05 07:17:01 echidna wicked[1279]: enp4s0f1 enslaved Jan 05 07:17:01 echidna wicked[1279]: enp4s0f2 setup-in-progress Jan 05 07:17:01 echidna wicked[1279]: enp0s31f6 setup-in-progress Jan 05 07:17:01 echidna wicked[1279]: br0 up Jan 05 07:17:01 echidna wicked[1279]: bond0 enslaved Created attachment 864293 [details]
Test script: bond-ip.sh [-1 ethX -2 ethY] [-m mode] [one|two|]
This script sets up a bonding to illustrate the behavior/rtnetlink enslave
processing change in 6.1 kernel -- regression to 4.x (sle12) - 6.0.x kernels.
It is using the `ip link` commands, stop/shutdown (bond0 in) wicked before.
Use the optional -1 and -2 options to specify the slave interface names;
it is setting up a bond0 in active-backup mode to work with any switch.
* bond-ip.sh -1 ethX -2 ethY one
Is the variant wicked is using and setting the link down and enslaving to
bond in a single `ip link set down master bond dev ethX` call aka rtnetlink
message that were working with all 4.x .. 6.0.x kernels and stopped to work
with 6.1 kernel.
[¹] A slave has to be set down before enslave or the kernel will reject it.
* bond-ip.sh -1 ethX -2 ethY two
Possible workaround -- as two instead of one `ip link set` rtnetlink message
any more, thus it _may_ cause some race conditions or side effect...
The 1st ip link is setting the slave down, 2nd enslaves to the master.
* bond-ip.sh [-1 ethX -2 ethY]
This is a [¹] setup failure case without to set the links down before enslave
to show that setting/ensuring that the slave links are down is required.
Created attachment 864294 [details]
Test script outputs with 4.x, 6.0 and 6.1 cases
$ grep ^=== bond-ip.txt
=== Working case "one" with SLE-12-SP5 4.x kernel ======================================
=== Working case "two" with SLE-12-SP5 4.x kernel ======================================
=== Failure case "fail" with SLE-12-SP5 4.x kernel ======================================
=== Regression case "one" with Tumbleweed 6.1 kernel ===================================
=== Working case "two" with Tumbleweed 6.1 kernel ======================================
=== Failure case "fail" with Tumbleweed 6.1 kernel ======================================
=== Working case "one" with Tumbleweed 6.0 kernel ======================================
=== Working case "two" with Tumbleweed 6.0 kernel ======================================
=== Failure case "fail" with Tumbleweed 6.0 kernel =====================================
(In reply to Andrei Borzenkov from comment #5) > (In reply to Alessandro Sturniolo from comment #4) > > kernel-default-6.0.10-1.1.x86_64 > > kernel-default-6.1.0-1.1.x86_64 > > Yes, I can reproduce it with kernel 6.1.1 and the problem goes away when I > boot kernel 6.0.3 without any change in configuration. Apparently something > changed between kernel 6.0 and 6.1. Whether it is a bug that must be fixed > in kernel or behavior change that must be fixed in wicked I do not know. Exactly. The enslave `ip link set down master bond0 dev ethX` in a single call works with all kernels since 4.x .. testsuite worked on SLE-12-SP2, I just don't have it installed on my test machine any more, thus 12-SP5 in the test outputs above. So this is a question to out kernel-maintainers: is this a bug or feature? We could change to send two separate messages, but it would be better when the kernel would get a fix to work with one down+enslave message again as sending two isn't "atomic" any more. (In reply to Marius Tomaschewski from comment #11) > > So this is a question to out kernel-maintainers: is this a bug or feature? > You can't please 'em all ... commit a4abfa627c3865c37e036bccb681619a50d3d93c Author: Phil Sutter <phil@nwl.cc> Date: Wed Sep 14 17:06:23 2022 +0200 net: rtnetlink: Enslave device before bringing it up Unlike with bridges, one can't add an interface to a bond and set it up at the same time: | # ip link set dummy0 down | # ip link set dummy0 master bond0 up | Error: Device can not be enslaved while up. Of all drivers with ndo_add_slave callback, bond and team decline if IFF_UP flag is set, vrf cycles the interface (i.e., sets it down and immediately up again) and the others just don't care. Support the common notion of setting the interface up after enslaving it by sorting the operations accordingly. > We could change to send two separate messages, but it would be better when > the kernel would get a fix to work with one down+enslave message again as > sending two isn't "atomic" any more. Well ... I have a feeling that as long as interface must be down to enslave it this commit indeed opens up race condition without really fixing anything. > ip link set dummy0 master bond0 up
This is IMO a faulty call for a bond. bond != bridge
Note also: it is IMO not a good idea to set the slaves UP yourself, e.g.:
ip link set master bond0 dev ethX ## enslave to bond0 (must be down)
ip link set up dev ethX ## "hack" to set the slave UP after enslave
The interface needs to be down while enslave (set master). The bonding
driver is not only setting it up itself, but also performs several
actions on the slaves (inherits MTU, MAC, vlan filter, ... offloads).
When we set the slave UP after enslave, we'll most probable disturb
or break these actions -- some nic drivers may even reset the HW
(and remove the administrative UP flag .. at least for a while) when
they're applying offloads... ==> they may need the slave in down state
(especially the expensive, full featured offload NICs IMO tend to
behave like this).
Andrei, I've prepared a workaround for this kernel behavior regression to not set the DOWN flag any more in wicked bond enslave: https://github.com/openSUSE/wicked/pull/952 Once our CI tests passed, we'll apply it to: https://download.opensuse.org/repositories/network:/wicked:/testing/ repo https://download.opensuse.org/repositories/network:/wicked:/testing/ kernel 6.0.12 - bonding working kernel 6.1.8 - not working It never arrived to the testing branch project -- seems, it was outdated there (still 0.6.71) ... whatever the reason were [CI hackweek?]. Sorry! I've manually prepared a test package with pull/952 applied as patch: https://download.opensuse.org/repositories/home:/mtomaschewski:/branches:/network:/wicked:/master/ * Tue Feb 07 2023 mt@suse.com - Applied pull request https://github.com/openSUSE/wicked/pull/952 to workaround kernel 6.1 enslave processing regression breaking `ip link set down master bond0 dev eth0` like rtnetlink enslave, that worked with all kernels from 4.12 up to 6.0 (boo#1206674). [+ 0001-bond-workaround-6.1-enslave-regression-boo-1206674.patch] repo https://download.opensuse.org/repositories/home:/mtomaschewski:/branches:/network:/wicked:/master/ wicked-0.6.72-20.1 kernel 6.0.12 - bonding working kernel 6.1.8 - bonding working (In reply to Alexey Bobrov from comment #17) > repo > https://download.opensuse.org/repositories/home:/mtomaschewski:/branches:/ > network:/wicked:/master/ > > wicked-0.6.72-20.1 > kernel 6.0.12 - bonding working > kernel 6.1.8 - bonding working Thanks for retesting / verifying it! The pull is accepted to master https://github.com/openSUSE/wicked/pull/952 now (for next release), applied as patch to network:wicked:factory/wicked and submitted to Factory: https://build.opensuse.org/request/show/1064159 SUSE-RU-2023:2747-1: An update that has five recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1194557, 1203300, 1206674, 1211026, 1211647 Sources used: openSUSE Leap 15.5 (src): wicked-0.6.72-150500.3.7.1 Basesystem Module 15-SP5 (src): wicked-0.6.72-150500.3.7.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. SUSE-RU-2023:2875-1: An update that contains one feature and has six recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1194557, 1203300, 1206447, 1206674, 1206798, 1211026 Jira References: PED-90 Sources used: SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1 (src): wicked-0.6.73-150100.3.32.1 SUSE Linux Enterprise Server 15 SP1 LTSS 15-SP1 (src): wicked-0.6.73-150100.3.32.1 SUSE Linux Enterprise Server for SAP Applications 15 SP1 (src): wicked-0.6.73-150100.3.32.1 SUSE CaaS Platform 4.0 (src): wicked-0.6.73-150100.3.32.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. |