Bug 1188316

Summary:	Configuring multiple default gateways in wicked via yast result in unpredictable routing
Product:	[openSUSE] openSUSE Tumbleweed	Reporter:	Robert Mahar <bob>
Component:	Network	Assignee:	wicked maintainers <wicked-maintainers>
Status:	RESOLVED FEATURE	QA Contact:	E-mail List <qa-bugs>
Severity:	Normal
Priority:	P5 - None	CC:	mt
Version:	Current
Target Milestone:	---
Hardware:	x86-64
OS:	openSUSE Tumbleweed
Whiteboard:
Found By:	---	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---

Description Robert Mahar 2021-07-14 21:03:55 UTC

Found on Tumbleweed 20201108 same seem to happen on latest version I tested ( 20210712 )

Steps to reproduce:

3 NICS on different IPV4 networks
In YAST with Wicked selected ...
configure a static IP on each interface in its respective IP networks.   For example:

10.10.10.10 / 24
192.168.11.11 / 24
172.16.12.12 / 24

In each network is x.x.x.1 which is an external router to the Internet.

Configuring a default gateway for ONE of these interfaces pointing at its corresponding x.x.x.1 gateway address results in the expected outcome and the kernel routing table shows the routes to the locally attached networks and the default rout to x.x.x.1, for example 10.10.10.1

If TWO default routes are created in Yast, only one appears in the routing table, seemingly the "first" one.  

However you find that

test:/etc/sysconfig/network # cat ifroute-ens???

default 10.10.10.1 - ens192 
default 192.168.11.1 - ens224

also 

test:/Build # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.10.10.1      0.0.0.0         UG    0      0        0 ens192
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 ens192
192.168.11.0    0.0.0.0         255.255.255.0   U     0      0        0 ens224
172.16.12.0     0.0.0.0         255.255.255.0   U     0      0        0 ens160

if you shutdown one or the other interface using

wicked ifdown ensNNN

it is seen that the default route of the surviving interface is not populated in the kernel routing table.   

test:/Build # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 ens192
192.168.11.0    0.0.0.0         255.255.255.0   U     0      0        0 ens224
172.16.12.0     0.0.0.0         255.255.255.0   U     0      0        0 ens160

on bringing up the same interface, the default route is populated for the interface.   Whatever interface was shut down and brough up last has its entry as the one and only default route.

test:/Build # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.10.10.1      0.0.0.0         UG    0      0        0 ens192
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 ens192
192.168.11.0    0.0.0.0         255.255.255.0   U     0      0        0 ens224
172.16.12.0     0.0.0.0         255.255.255.0   U     0      0        0 ens160

manually adding the route

route add -net 0.0.0.0 netmask 0.0.0.0 gw 192.168.11.1

yields the expected outcome:

test:/Build # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.10.10.1      0.0.0.0         UG    0      0        0 ens192
default         192.168.11.1    0.0.0.0         UG    0      0        0 ens224
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 ens192
192.168.11.0    0.0.0.0         255.255.255.0   U     0      0        0 ens224
172.16.12.0     0.0.0.0         255.255.255.0   U     0      0        0 ens160

So in all honesty I don't know where it going wrong or if this is working as designed, but the design is not aligned with the needs of dual homed systems.   Yast seems to be populating the route config files, and wicked does seem to activate only one of the default routes configured.  The kernel is happy to accept several default routes when done manually.

Three NICS is not necessary to observe the problem described.  The essential configuration is having multiple default gateways configured in Yast.

Comment 1 Robert Mahar 2021-07-14 21:09:07 UTC

NB: In the report I missed the last line:
- - - -

test:/etc/sysconfig/network # cat ifroute-ens???

default 10.10.10.1 - ens192 
default 192.168.11.1 - ens224
default 172.16.12.1 - ens160  <-- 

- - - -

all three interfaces show a default route configured when looking at the files.

Comment 2 Marius Tomaschewski 2021-08-10 07:51:55 UTC

This setup does not work properly.

Routes for same destination + metric (+ tos) are conflicting routes,
use policy routing to configure this properly - without yast2, which
does not support such setups (thus not a bug, but a not implemented
feature):

ifcfg-ens192:
   IPADDR=10.10.10.10/24
ifroute-ens192:
   10.10.10.0/24    -            - - table 192 src 10.10.10.10
   default          10.10.10.1   - - table 192
   # Further: choose an interface to have route in the main table:
   default          10.10.10.1
   # or use a different metric on each interface, e.g. 10 here:
   #default          10.10.10.1   - - metric 10
ifrule-ens192:
   ipv4 from 10.10.10.10 lookup 192

ifcfg-ens224:
   IPADDR=192.168.11.11/24
ifroute-ens224:
   192.168.11.0/24  -            - - table 224 src 192.168.11.11
   default          192.168.11.1 - - table 224
   #default          192.168.11.1 - - metric 20
ifrule-ens224:
   ipv4 from 192.168.11.11 lookup 224

ifcfg-ens160:
   IPADDR=172.16.12.12/24
ifroute-ens160:
   172.16.12.0/24   -            - - table 160 src 172.16.12.12
   default          172.16.12.1  - - table 160
   #default          192.168.11.1 - - metric 30
ifrule-ens160:
   ipv6 from 172.16.12.12 lookup 160

Instead of "table 160" and "lookup 160", you can define names
for the routing tables in /etc/iproute2/rt_tables.

See also:
  https://lartc.org/howto/lartc.rpdb.multiple-links.html#AEN267

Comment 3 Marius Tomaschewski 2021-08-10 08:02:25 UTC

(In reply to Marius Tomaschewski from comment #2)
> ifroute-ens160:
>    172.16.12.0/24   -            - - table 160 src 172.16.12.12
>    default          172.16.12.1  - - table 160
>    #default          192.168.11.1 - - metric 30
                       ^^^^^^^^^^^^
there was typo:        172.16.12.1