Bug 1096142

Summary: immutable /etc/resolv.conf blocks offline upgrade to 15.0
Product: [openSUSE] openSUSE Distribution Reporter: uli fuerst <ulifue>
Component: InstallationAssignee: Ladislav Slezák <lslezak>
Status: RESOLVED FIXED QA Contact: Jiri Srain <jsrain>
Severity: Major    
Priority: P3 - Medium CC: astieger, jreidinger, kanderssen, mrmazda
Version: Leap 15.0   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE 42.3   
URL: https://trello.com/c/AWSdctvi
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: save_y2logs file

Description uli fuerst 2018-06-06 03:57:31 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Build Identifier: 

Right at the beginning of the upgrade, after inspecting the disk I get the window titled "Partition or System to upgrade" and it shows the root partition. After clicking the "Next" button I get the message "mounting partition". After a while I get the following error message:
Error
Internal Error. Please report a bug report with logs. Run save-y2logs to get complete logs.
Details: Operation not permittet @ rb_sysopen - /mount/etc/resolv.conf
Caller: /mounts/mp_0001/user/lib64/ruby/2.5.0/fileutils.rb: 1292: in `initialize'
Start the Ruby debugger now to debug the issue? (Experts only)

Reproducible: Always

Steps to Reproduce:
1. Insert the install DVD 
2. Reboot
3. Follow instructions on screen for upgrade
Actual Results:  
I get the error message as described above

Expected Results:  
Root partition is mounted and install/upgrade proceeds

I have saved the y2log file but I cannot see a way to attach it.
Comment 1 Andreas Stieger 2018-06-06 05:04:32 UTC
*** Bug 1096143 has been marked as a duplicate of this bug. ***
Comment 2 Andreas Stieger 2018-06-06 05:05:15 UTC
*** Bug 1096145 has been marked as a duplicate of this bug. ***
Comment 3 Andreas Stieger 2018-06-06 05:08:20 UTC
(In reply to uli fuerst from comment #0)
> I have saved the y2log file but I cannot see a way to attach it.

Use the "Add Attachment" option.
Comment 4 uli fuerst 2018-06-06 05:23:15 UTC
The attachment is over 80KB so therefore the connection refused. That was the reason for the duplicates of this bug - I thought the bug report did not get through either. Sorry about that.
Comment 6 uli fuerst 2018-06-07 03:03:47 UTC
I still don't know how to attach a 80Kb y2log-...tar.xz file and I don't know if it is possible to find the problem without the log file.
Comment 7 Andreas Stieger 2018-06-07 07:10:17 UTC
80 KiB should be no problem for bugzilla. Hmm https://xkcd.com/949/
If that does not work for you locally please find an alternative.
Comment 8 uli fuerst 2018-06-07 22:23:53 UTC
Created attachment 773067 [details]
save_y2logs file
Comment 9 uli fuerst 2018-06-07 22:26:22 UTC
Must have been a problem with my computer. It worked from a different one to send the attachment.
Comment 10 Andreas Stieger 2018-06-08 07:08:46 UTC
clearing needinfo flag
Comment 11 Ladislav Slezák 2018-06-08 14:59:10 UTC
I found out that it crashes when YaST wants to copy /etc/resolv.conf from the inst-sys to /mnt (https://github.com/yast/yast-update/blob/75eaa6da6bf61188c7864a7d4e3cf9ebf6578eec/src/modules/RootPart.rb#L1722)

The error message is: Operation not permitted @ rb_sysopen - /mnt/etc/resolv.conf' (Errno::EPERM).

Do you use some special permissions or options for /etc (or /) in the system? What's the old system you are upgrading? Or any idea why that file could not be written?
Comment 12 Felix Miata 2018-06-08 16:46:48 UTC
Bug reporters are CC'd whether they want to be or not in all versions of bugzilla software since the beginning of time. Expressly adding the reporter's email address to the CC list is unnecessary conflation.
Comment 13 uli fuerst 2018-06-08 22:12:01 UTC
AS mentioned in the report, the system is Leap42.3, I didn't change any permissions so I presume they are standard. Here are the results or ls -l for /etc and /etc/resolv.conf. Hope that helps
drwxr-xr-x 149 root root   20480 Jun  8 12:26 etc
-rw-r--r--  1 root root       162 Dec 21  2016 resolv.conf
Comment 14 uli fuerst 2018-06-08 22:25:11 UTC
Something is strange - I have a whole lot of files /etc/resolv.conf, here are a few from the le -l command:
-rw-r--r--  1 root root       162 Dec 21  2016 resolv.conf
-rw-r--r--  1 root root       807 Oct 28  2012 resolv.conf.0bwWRT
-rw-r--r--  1 root root       807 Sep 27  2012 resolv.conf.0NRBtG
-rw-r--r--  1 root root       807 May 18  2012 resolv.conf.0QwUrc
-rw-r--r--  1 root root         0 Jan 31  2013 resolv.conf.1fu66V
-rw-r--r--  1 root root       807 May  6  2012 resolv.conf.2eVqir
-rw-r--r--  1 root root       807 Jan 22  2013 resolv.conf.2SBgUK
-rw-------  1 root root         0 Aug  1  2013 resolv.conf.3FXkqm
-rw-------  1 root root         0 Feb  7  2013 resolv.conf.4j7JUA
-rw-r--r--  1 root root       807 Dec 20  2012 resolv.conf.5AFNu5
-rw-r--r--  1 root root       849 Jun 18  2012 resolv.conf.5PJszp
-rw-r--r--  1 root root       807 Jan 14  2013 resolv.conf.6RcCWb
-rw-r--r--  1 root root         0 Jun 10  2013 resolv.conf.8cKkji
It goes on for a while, the current file is:
:~> cat /etc/resolv.conf
# IPv4 name servers OpenDNS:
nameserver 208.67.222.222
nameserver 208.67.220.220

# IPv6 name servers OpenDNS:
nameserver 2620:0:ccc::2
nameserver 2620:0:ccd::2
Comment 15 Felix Miata 2018-06-09 02:22:54 UTC
re: "a few" - Do any of them have recent or current timestamps? They shouldn't.

If you can reproduce this:

		# lsattr /etc/resolv.conf
		----i--------e-- /etc/resolv.conf

(It's the "i" that matters -> it shouldn't be there.)

then you need to do this:
		# chattr -i /etc/resolv.conf

to produce ~this:
		# lsattr /etc/resolv.conf
		-------------e-- /etc/resolv.conf

All those 2012 and 2013 files can be deleted. Config files in such form as those are created as part of an incomplete upgrade process, one that prohibited writing to the original file.
Comment 16 uli fuerst 2018-06-09 21:36:07 UTC
Thanks, Felix, none have recent timestamp and they are deleted now, and the chattr -i command did the trick. Afterwards the upgrade went without a hitch!
Comment 17 Ladislav Slezák 2018-06-11 09:38:36 UTC
(In reply to Felix Miata from comment #15)
> If you can reproduce this:
> 
> 		# lsattr /etc/resolv.conf
> 		----i--------e-- /etc/resolv.conf
 

Ah, good point Felix! So the file attribute was set to "immutable"!

The question is who set this attribute? Comment #13 suggests it was not changed manually.


On my older 42.3 testing system the file has no extra attributes:

# lsattr /etc/resolv.conf 
---------------- /etc/resolv.conf

I installed all online updates and the attributes were not changed. Even switching from the Wicked backend to the Network Manager did not change the attributes.

Any idea how that was changed? Is that a common problem?
Comment 18 Ladislav Slezák 2018-06-11 10:43:02 UTC
(In reply to uli fuerst from comment #14)

Something is strange - I have a whole lot of files /etc/resolv.conf, here are a few from the le -l command:
-rw-r--r--  1 root root       162 Dec 21  2016 resolv.conf
-rw-r--r--  1 root root       807 Oct 28  2012 resolv.conf.0bwWRT

I guess there is some script or service running in background updating the /etc/resolv.conf and sometimes it left a temporary file (or backup).

> It goes on for a while, the current file is:
> :~> cat /etc/resolv.conf
> # IPv4 name servers OpenDNS:
> nameserver 208.67.222.222
> nameserver 208.67.220.220

So you are using the OpenDNS server for name resolution. How did you set that up?
Comment 19 Felix Miata 2018-06-11 15:22:50 UTC
/etc/resolv.conf.netconfig.s122.big41.01 from openSUSE 12.2 on host big41:

### /etc/resolv.conf file autogenerated by netconfig!
#
# Before you change this file manually, consider to define the
# static DNS configuration using the following variables in the
# /etc/sysconfig/network/config file:
#     NETCONFIG_DNS_STATIC_SEARCHLIST
#     NETCONFIG_DNS_STATIC_SERVERS
#     NETCONFIG_DNS_FORWARDER
# or disable DNS configuration updates via netconfig by setting:
#     NETCONFIG_DNS_POLICY=''
#
# See also the netconfig(8) manual page and other documentation.
#
# Note: Manual change of this file disables netconfig too, but
# may get lost when this file contains comments or empty lines
# only, the netconfig settings are same with settings in this
# file and in case of a "netconfig update -f" call.
#
### Please remove (at least) this line when you modify the file!
nameserver 8.8.8.8

It is probably similar to most of those extra resolv.conf.* files in comment 14, while the zero byte balance of those others suggest name resolution went through a period of trouble in 12.2, which was solved six years ago, and no longer remembered, by locking down a working resolv.conf with the immutable attribute. Online upgrades since then could have been successful, and only now offline blocked.
Comment 20 uli fuerst 2018-06-11 21:23:31 UTC
Just to clarify what I have done:
1) at some stage I set the network configuration from wicked to network manager, simply because I want to see a confirmation that the network is up in the system tray. Wicked does not show any icons there.
2) Some problems (slowness) with the dns servers from my ISP caused me to change to opendns. I changed it in YAST to manually and entered the dns servers for IPv4 and IPv6 manually in /etc/resolv.conf. (Yast hadn't enough fields for IVv4 and IPv6)
3) When visiting my daughter I could not get onto the internet as their ISP only accepts their own dns servers and nothing else. (I had an IP address but the nslookup command timed out as there was no contact to a dns server). So I either had to change back Yast to autoconfig (or however this is called) or I had to manually enter their ISP's dns servers. This switching backwards and forwards happened at least twice a year.

I don't know what set the attribute to immutable but the above goes back at least 5 years (may be back to version 12.2 as Felix suggests?). During this time I have only upgraded on this computer.
Comment 21 Ladislav Slezák 2018-06-12 12:55:21 UTC
Oh, upgrading since 12.2... That is quite rare scenario I guess so I do not expect many affected users with this issue.

On the other hand I googled a bit and found out that the immutable trick was quite often suggested as a solution to avoid overwriting the file.

I'll add handling the Errno::EPERM exception into the code...
Comment 22 Knut Alejandro Anderssen González 2018-06-14 09:47:20 UTC
Ladislav, taking in account #21 I will assign it directly to you. Feel free to reassign it to yast2-maintainers in case you prefer to create a Trello Card adding it to our Incoming board in order to prioritize it.
Comment 23 Ladislav Slezák 2018-06-19 12:08:36 UTC
Just for the record WIP PR: https://github.com/yast/yast-update/pull/103
Comment 24 Josef Reidinger 2018-08-21 12:19:52 UTC
fix merged.