Bug 1164010

Created attachment 830394 [details]
contents of /var/log/zypp/history

Created attachment 830395 [details]
contents of /var/log/messages-20200216.xz  file

Created attachment 830396 [details]
contents of /var/log/zypper-20200216.xz

Created attachment 830404 [details]
journalctl -a output

Created attachment 830405 [details]
output of the command ls -al /var*

Created attachment 830406 [details]
ls -al /var/lib/*   output

Hello,

My PC has dual boot Linux/Windows .
Today I tried to start Win but by mistake a selected Linux .
It started to display some failure then entered in Maint mode. 
I was able to run journalctl -a and collect the data, as well as the dirs files 
of /var and /var/lib  .
Is there a way to start the linux in Maintenance mode ?
If so I might be possible that PC enters in main mode close to ...
Starting Hold until boot process finishes up
Starting terminate Plymouth Boot Screen .

Please let me know .


Thank you for your help.

ND

very likely kernel-firmware again...

Try the kernel in OBS Kernel:openSUSE-15.1 repo.  If it's a known issue like iwlwifi problem, it should have been already addressed there.

Hello Everyone ,
 
I do not have an operational Linux environment . 
The only path to do something is via rescue.  
yast2 comes up. zypper lr works, but zypper ref fails .
So I am lost .... how could I try the OBS Kernel:openSUSE-15.1 repo . 
What kernel / level  ? 
Could you please give me a work around to be able to handle packages ?
What I can do is to get a package from opensuse repo using Windows .
How could I install that package in linux started via rescue ?
 
Could you please advise me how to proceed ?

Thank you,
ND

Get kernel-default.rpm from OBS download URL for the given project, i.e.
  http://download.opensuse.org/repositories/Kernel:/openSUSE-15.1/standard/

And install it from the rescue system.

Or you can try blacklisting iwlwifi module at first (supposing that's the cause).
Just boot with the kernel option:
  module_blacklist=iwlwifi

If this can boot up properly, it confirms that the problem is iwlwifi, and this should have been fixed in the recent openSUSE-15.1 kernel.

Then copy the already downloaded kernel-firmware*.rpm via USB stick or whatever method, and install it via
  zypper in -f kernel-default*.rpm

... and reboot to that kernel by choosing in GRUB menu.

Note that this kernel won't for in Secure Boot mode.

Hello, 
Thank you for the information .
I got the on  
kernel-default-base-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
packages from 
http://download.opensuse.org/repositories/Kernel:/openSUSE-15.1/standard/
then I tried to install them from the rescue system .
yast2 is coming up, but except listing repos everything fails :
software --> Cannot read the list of installed packages 

online update --> Cannot access installation media (Apache repo)
curl for https://.... connection failed 
Could not resolve host  download.opensuse.com

except zypper lr any other commands fail -->
zypper ref ; zypper se iwlwifi ; 
zypper in <path>/kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
All attempts fail due to :  Target initialization failed: Rpm exception

Tried to see if iwlwifi exists  lsmod | grep iw  is empty 
[ however lsmod displays  a lot of lmods ) 


I believe that the any option is trying to boot with the kernel option 
module_backlist=iwlwifi 
I retired 7 years ago (MVS sysprog) and I forgot many of Linux  kernel, GRUB settings/commands via bootloader or at boottime 

Could you please tell me how this is done ?

Thank you,
ND   
many things in Linux area .

(In reply to Nick Dordea from comment #12)
> Hello, 
> Thank you for the information .
> I got the on  
> kernel-default-base-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
> kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
> packages from 
> http://download.opensuse.org/repositories/Kernel:/openSUSE-15.1/standard/

First off, you don't need kernel-default-base*.rpm.  Only kernel-default-4.12.*.rpm is needed.

> then I tried to install them from the rescue system .
> yast2 is coming up, but except listing repos everything fails :
> software --> Cannot read the list of installed packages 

You can't do it for the rescue system in that way...
You need to mount the device manually, then zypper or rpm install with root option.  But I'm afraid it's too much unless you understand how the rescue system works.

> I believe that the any option is trying to boot with the kernel option 
> module_backlist=iwlwifi 
> I retired 7 years ago (MVS sysprog) and I forgot many of Linux  kernel, GRUB
> settings/commands via bootloader or at boottime 

This one should be easy, OTOH.
On GRUB boot menu, press 'e' button.  It'll open a window to allow you edit.
Go down to the line "linux /boot/vmlinuz..." and append "module_blacklist=iwlwifi" at the end of the line, then press F10 or Ctrl-X to boot.
Some details are found in the online documentation:
  https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.reference/cha-grub2.html
the section "12.2.5 Editing Menu Entries during the Boot Procedure".

Hello,
Thank you for your comments and info .
I would like to explain what I have done in rescue mode, 
On August 2019 I upgraded to leap 15.1 using the CD-Network it worked fine.
For this issue, to go in the rescue mode I used the same CD ...
-Boot from CD
after that I mounted the root partition as well as /proc, /dev and /sys partitions as follows [ based on the documentation of opensuse leap 42.x / 15.0 ]
mount /dev/sda2 /mnt
mount -t proc none /mnt/proc
mount --rbind /dev /mnt/dev
mount --rbind /sys /mnt/sys

Then 
chroot /mnt /bin/bash 
mount -a 

At this point I have access to the installed system .
In this environment I tried to use yats2, zypper  detailed in my previous comment.
I used this procedure several times fix file systems issues, settings, etc .

If this procedure is not correct please let me know .

I will try to boot then blacklist iwlwifi module . I am aware that this new boot is not permanent .

I will keep you posted with the kernel option try ...

Thank you,
ND

Hello,

Tried a boot with the kernel option module_blacklist=iwlwifi
Like explained, module_blacklist=iwlwifi was appended to 
linux /boot/vmlinuz ...  command 

The boot failed in the same manner as before .
Messages that services failed   
...
Starting Terminate Plymouth Boot Screen
Starting Hold until boot process finishes up

Then the system is frozen .

Does the journalctl output point to other culprits  [ rf kill , etc ] ?

Could we install the kernel 4.12.4-lp151.102.1 by using rpm instead of zypper ?
I'm going to try installing it with rpm ... 

At this point, is reinstalling the distribution leap 15.1 [ August 2019 ] something to be considered ? 

Thank you,
ND

Hello, 

I tried to install kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm 
via rpm from rescue env  created as per previous comment .

rpm -i -h -v --test http://download.opensuse.org/repositories/Kernel:/openSUSE-15.1/standard/kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
failed 
curl: (6) Could not resolve host download.opensuse.org

rpm -i -h -v --test <path>/kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
has a warning <path>/kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm Header V3 RSA/SHA256 Signature Key ID 03579cad NOKEY
but completed ok 

rpm -i -h -v <path>/kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm
installed fine and created another boot entry 
Created initrd /boot/initrd/.....kernel-default-4.12.14-lp151.102.g35fcd79.default

A) Try to boot with the new kernel ....
selecting the above boot entry ...

Unfortunately it does not solve the problem . Got the failures then it is frozen 


B) rpm -q kernel-firmware  shows 
kernel-firmware-20200107-lp151.2.12.1.noarch

rpm -l kernel-firmare shows a lot of entries like 
/lib/firmware/iwlwifi*

again lsmod | grep iwl  returns a null answer .
How could I check the status of this lmod ? Is it installed [ modprobe  ] ?

Questions :
1. Is the new kernel needed anymore ?  Should I remove it via rpm ?
2. Is the kernel-firmware-20200107-lp151.2.12.1.noarch needed ?

Maybe the changes due to the new kernels affects the Network Manager and the NM  triggers a domino effect on NM dependent services ?

As I said in one of my prior comments  is reinstalling the leap 15.1 doing any good ? Doing that will wipe out all the maintenance applied and will regress my machine  by 6-7 months ago .

Please tell me what is your opinion .

Thank you,
ND

(In reply to Nick Dordea from comment #15)
> Hello,
> 
> Tried a boot with the kernel option module_blacklist=iwlwifi
> Like explained, module_blacklist=iwlwifi was appended to 
> linux /boot/vmlinuz ...  command 
> 
> The boot failed in the same manner as before .
> Messages that services failed   
> ...
> Starting Terminate Plymouth Boot Screen
> Starting Hold until boot process finishes up
> 
> Then the system is frozen .

OK, then try to boot with the following boot option instead:

  nomodeset

This will disable the native graphics.

If this worked, please try to downgrade the kernel-firmware package.
You can see the list of

  zypper se -s kernel-firmware

and install the older version by specifying the old version and with -f option of zypper install, e.g.:

  zypper in -f kernel-firmware-20191118-lp151.2.9.1

Then reboot without extra boot option again.

(In reply to Nick Dordea from comment #16)
> I tried to install kernel-default-4.12.14-lp151.102.1.g35fcd79.x86_64.rpm 
> via rpm from rescue env  created as per previous comment .

I guess operations you took were all insufficient.  For installing / recovering from the rescue image, you'd need to mount the filesystem manually and locally, and install there with specific option.  Judging from your description, the whole those procedures seem skipped, so it makes no use if you do zypper install or whatever.

Please try to boot with the existing kernel with some workaround boot option at first, and forget about the rescue system.

In anyway, I'll be off from now, so someone else needs to take care.  I reassign back to kernel-maintainers.

Hello, 
I tried to boot  leap 15.1  14.12.14-lp151-28.36  with boot option nomodeset 
All failures [ Network Manager, Manage Install and create color profiles, MySQL, Clamav, Freshclam , rf kill etc ] happens again .
At the end I got a blank screen . Except moving the mouse nothing can be done .
I believe that last test failed ... it not give me access to linux environment.
----------------

As I explained before, I do not have access to my Linux machine .
I tried to do some work booting in rescue mode .
I followed a opensuse doc on how to get access to the installed system via 
booting from a linux distribution CD and selecting rescue option .

Here are the steps I have followed :
1. boot from the CD leap-15-1 

2.  userid root

3. lsblk to select the root partition [ / ]
   It is /dev/sda2

4.Manually Mount the essential partitions 
mount /dev/sda2  /mnt
mount -t proc none /mnt/proc
mount --rbind  /dev /mnt/dev
mount --rbind  /sys /mnt/sys

5. Change root into new env
chroot /mnt /b in/bash 

6. Mount the remaining partition
mount -a 

The doc says that at this time I have access to installed system .

Behavior
A.
except zypper lr, everything else fails with 
Target initialization failed: Rpm exception 

B.
yast2 is coming up
repositories --> I can see the installed/defined repos 
SW management --> fails  Can not read the list of installed packages
OnLine Update --> Cannot access installation media 
              Apache repo 
              curl for /https:// (apache repo ) connection failed 
              could not resolve host download.opensuse.org
sysconfig ---> seems ok 
I did not take any risks to go in other yast2 areas .

C.
I tried to use rpm and it is working 
rpm -q kernel-default kernel-firmware are detected 
as well other installed packages ( mariadb, etc )

-------------------


I was told "You need to mount the device manually, then zypper or rpm install with root option .  But I'm afraid it's too much unless you understand how the rescue system works. " 

Could you please tell me if the above procedure to get in the rescue environment is valid . If not could you please point me to a doc I can follow .

Using the rescue option is the only way to get into the installed system .

I believe that the last maintenance---that mentioned changes incore libraries and kernel-firmware---changed something that affects essential areas :  Network Manager, Logon screen[sddm?,plasma?] .  
The first failures I see relate to Load/Save RF Kill Switch Status and 
 systemd-rkfill.service and aparmor ...

Could you please have a look at some of my attachments [ journalctl output, 
/var/zypp/logs, history and info re /var structure ] and tell me if the system can be fixed from a "good" rescue system ?

If that info is not enough please tell me what&how to collect .

Thank you,
ND

Hello, 
Yesterday I did some analysis of journalctl outputs and more testing 
A) journalctl autput attacched at the time of opening the bug 1164010 ]
Feb 17 18:11:07 suse4Rocco systemd-rfkill[936]: Failed to create rfkill directory: Not a directory
Feb 17 18:11:07 suse4Rocco systemd-random-seed[938]: Failed to create directory /var/lib/systemd/: Not a directory
Feb 17 18:11:07 suse4Rocco systemd-backlight[940]: Failed to create backlight directory /var/lib/systemd/backlight: Not a directory
Feb 17 18:12:26 suse4Rocco systemd-tmpfiles[1168]: Failed to create directory or subvolume "/var/lib/PackageKit": Not a directory 

B) after the system appeared frozen I tried to see if Alt-Ctrl-F1 works ....
Eureka !  I got access to the installed linux [ without rescue]
so I had possibility to get a full journalctl output .
[file journalctl-a_console-mode.txt]
I had a look at its output and I noticed lots of messages compalining that 
/var/lib/xxxxxx Not a directory 
some examples below :
Feb 19 16:10:06 suse4Rocco systemd-rfkill[972]: Failed to create rfkill directory: Not a directory
Feb 19 16:10:06 suse4Rocco systemd[1]: systemd-rfkill.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 16:10:06 suse4Rocco systemd[1]: Failed to start Load/Save RF Kill Switch Status.
...
Feb 19 16:10:15 suse4Rocco NetworkManager[1345]: Cannot create parents for '/var/lib/NetworkManager': Not a directory
...
Feb 19 16:10:26 suse4Rocco mysql-systemd-helper[1752]: Fatal error Can't create database directory '/var/lib/mysql'
...
Feb 19 16:10:30 suse4Rocco systemd[1]: Starting Manage, Install and Generate Color Profiles...
Feb 19 16:10:30 suse4Rocco kernel: nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
Feb 19 16:10:30 suse4Rocco start_apache2[1844]: AH00557: httpd-prefork: apr_sockaddr_info_get() failed for suse4Rocco
Feb 19 16:10:30 suse4Rocco start_apache2[1844]: AH00558: httpd-prefork: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Feb 19 16:10:31 suse4Rocco colord[1913]: CdMain: failed to load mapping database: Error creating directory /var/lib/colord: Not a directory
Feb 19 16:10:31 suse4Rocco systemd[1]: colord.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 16:10:31 suse4Rocco systemd[1]: Failed to start Manage, Install and Generate Color Profiles.
Feb 19 16:10:31 suse4Rocco systemd[1]: colord.service: Unit entered failed state.
Feb 19 16:10:31 suse4Rocco systemd[1]: colord.service: Failed with result 'exit-code'.
...

Feb 19 16:10:15 suse4Rocco systemd[1]: polkit.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 16:10:15 suse4Rocco systemd[1]: Failed to start Authorization Manager.
Feb 19 16:10:15 suse4Rocco systemd[1]: polkit.service: Unit entered failed state.
Feb 19 16:10:15 suse4Rocco systemd[1]: polkit.service: Failed with result 'exit-code'.
...

Then I recalled that such issues were the second day after maintenance [ journalctl autput attacched at the time of opening the bug 1164010 ]
Feb 17 18:11:07 suse4Rocco systemd-rfkill[936]: Failed to create rfkill directory: Not a directory
Feb 17 18:11:07 suse4Rocco systemd-random-seed[938]: Failed to create directory /var/lib/systemd/: Not a directory
Feb 17 18:11:07 suse4Rocco systemd-backlight[940]: Failed to create backlight directory /var/lib/systemd/backlight: Not a directory
Feb 17 18:12:26 suse4Rocco systemd-tmpfiles[1168]: Failed to create directory or subvolume "/var/lib/PackageKit": Not a directory
so all failing services described in the failure description .

C) Instead of /var/lib directory it was a /var/lib file .
Its contents is provided below 
-----BEGIN TRUSTED CERTIFICATE-----
MIIC+TCCAoCgAwIBAgINAKaLeSkAAAAAUNCR+TAKBggqhkjOPQQDAzCBvzELMAkG
A1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xKDAmBgNVBAsTH1NlZSB3
d3cuZW50cnVzdC5uZXQvbGVnYWwtdGVybXMxOTA3BgNVBAsTMChjKSAyMDEyIEVu
dHJ1c3QsIEluYy4gLSBmb3IgYXV0aG9yaXplZCB1c2Ugb25seTEzMDEGA1UEAxMq
RW50cnVzdCBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRUMxMB4XDTEy
MTIxODE1MjUzNloXDTM3MTIxODE1NTUzNlowgb8xCzAJBgNVBAYTAlVTMRYwFAYD
VQQKEw1FbnRydXN0LCBJbmMuMSgwJgYDVQQLEx9TZWUgd3d3LmVudHJ1c3QubmV0
L2xlZ2FsLXRlcm1zMTkwNwYDVQQLEzAoYykgMjAxMiBFbnRydXN0LCBJbmMuIC0g
Zm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxMzAxBgNVBAMTKkVudHJ1c3QgUm9vdCBD
ZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEVDMTB2MBAGByqGSM49AgEGBSuBBAAi
A2IABIQTydC6bUF74mzQ61VfZgIaJPRbiWlH47jCffHyAsWfoPZb1YsGGYZPUxBt
ByQnoaD41UcZYUx9ypMn6nQM72+WCf5j7HBdNq1nd67JnXxVRDqiY1Ef9eNi1KlH
Bz7MIKNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0O
BBYEFLdj5xrdjekIplWDpOBqUEFlEUJJMAoGCCqGSM49BAMDA2cAMGQCMGF52OVC
R98crlOZF7ZvHH3hvxGU0QOIdeSNiaSKd0bebWHvAvX7td/M/k7//qnmpwIwW5nX
hTcGtXsI/esni0qU+eH6p44mCOh8kmhtc9hvJqwhAriZtyZBWyVgrtBIGu4GMEIw
FAYIKwYBBQUHAwQGCCsGAQUFBwMBDCpFbnRydXN0IFJvb3QgQ2VydGlmaWNhdGlv
biBBdXRob3JpdHkgLSBFQzE=
-----END TRUSTED CERTIFICATE----- 
What program/product/package overrides /var/lib dir to use/create  var/lib file ?

D) Renamed the /var/lib file as /var/lib_file then created /var/lib_file
   
   Regress the kernels to the level of repo opensuse 15.1 update
   Take them from repo on a stick 
   zyppr in -f system <path>/kernel-default-4.12.14-lp151.28.36.1.x86_64.rpm <path>/kernel-firmware-20191118-lp151.2.9.1.noarch.rpm
   
   
E) reboot 
With the exception of :
Authorization Manager 
NetworkManager Wait On Line
freshclam 
Clam antivirus Demon
MySQL
Manage Install Generate Color Profiles
no other failures .

F) Force reinstall of packages related to those services ...
zypper in -f mariadb clamav  

G) After a new reboot the only issues are :
Authorization Manager  [ polkit.service ]
Manage Install Generate Color Profiles [ colord.service ]
NetworkManager Wait On Line
as user :
plasma does not start even if sddm.service is started
except autostarted programs [ krusader, yakuake ] 
other programs fail [ firefox, kontact, issues with starting wifi ] 
as root :
plasma appears ok and I have access to many functions 
including yast graphic mode .

H) As root, in yast2 graphic 
      1. protect kernel-firmware from being upgraded to 2020017 level
      2. select packages based on colord polkit NetworkManager Plasma gnome-color-manager etc.
      3. reinstall them . The installed OK
   Note :
   Many repositories has a problem ... [ Likely the system repos are ok ]
   Download (curl) error for 'https://download.opensuse.org/repositories/Apache/openSUSE_Leap_15.1/repodata/repomd.xml':
   Error code: Curl error 60
   Error message: SSL certificate problem: unable to get local issuer certificate
   
   
I) reboot 
   Only NetworkManager-waiting-on-line failed 
   All other services started ok 
   Logon as user ......
   Eyreka !!!!
   I'm back in business !
   
Questions :
1. Based on the info at C) above 
   What program/product/package overrides /var/lib dir to use/create  var/lib file ?
   
2. Do I need NetworkManager-waiting-on-line ?
   It would be OK to disable it ?

3. Any idea how to solve "Error message: SSL certificate problem: unable to get local issuer certificate" ?

Could you please help me to solve the above 3 Questions ?

Thank you, 
ND

Created attachment 830836 [details]
journalctl -a  contents  [ from console env ]

Hello,
In vain, I was waiting for any move in the bugs : 1164010,  1164586, 1164592.
The questions are not answered ... the issues are stillnot fully fixed .
No need for further wainting ...
As far as I am concerned , the bugs : 1164010,  1164586, 1164592 should be closed with reason .... dormant for almost a week .

ND

*** Bug 1164592 has been marked as a duplicate of this bug. ***

Now I'm back to work and processing the backlogs, then stumbled on this bug again.  It seems that the system got messed up by the forced update or so.

And now looking at the reports from the beginning, this might not be the kernel or kernel-firmware problem at all.  The known problematic issue with the firmware aren't found on your machine, and the description (the failure of starting services) don't match with the expected failure.

Is this rather an issue with apparomor or such?
In anyway, it's hard to judge how sane your system is right now.  If there is no /var/lib directory or such, it means that the file system got corrupted and it was lost.  This directory must be present from the beginning.

Hello , 
It is a pity that for a bug opened on 2020-02-17 I receive an answer on 2020-03-11 . 
Also it is a pity that I am told that 
"And now looking at the reports from the beginning, this might not be the kernel or kernel-firmware problem at all.  The known problematic issue with the firmware aren't found on your machine, and the description (the failure of starting services) don't match with the expected failure.
Is this rather an issue with apparomor or such?
In anyway, it's hard to judge how sane your system is right now.  If there is no /var/lib directory or such, it means that the file system got corrupted and it was lost.  This directory must be present from the beginning."  

It seems that nobody had a look at any of info I was able to collect from the failing machine . 

My comment #20  from 2020-02-20 I provided the file /var/lib that might-ve been the culprit of /var/lib destruction .  At that time I asked which component did that .  

... and many other 

It is a pity that the techical support, nonchalantly anounces me that kernel-firmware is not the cause as well as it seems that I am asked "Is this rather an issue with apparomor or such?" ?

As an retired MVS systprog I have found out that the way by this bug has been taken care of is strange and unexplicable .
Maybe the boots with different GRUB parameters, rather to fix the issue, they migth have been doing more damage to my system .

Thanks God that my machine is operational and I solved as much as possible.
It seems that all/some /var/lib objects are in lost+find [ their date is exactly that of initial crash ] . So definitely that maintenance had some issues which point to insufficient/lack of appropriate QA . 
To find out what was going on the  zypper logs were attached at bug opening ....

The only outstanding issues are :
1) NetworkManager-wait-online.service fails at boot .
I tried to find out what's wrong but the systemctl status messages and/or 
nmonline --help do not point to the cause ...
2) at shutdown I got the messafe failed to unmount /var 
I know that I was asking the Tech support about it and I was told that it is a known issue .   
Again .... is it failing unmount culprit ?

I believe that 1-2 weeks ago I asked the tech support to close this bug along with its "sisters" 1164586, 1164592 as not/partially fixed .
So, I ask again for theit closing as not/partially fixed .


Thanks,
ND

OK, let's close this one now and track in other bug entries.
BTW, for openSUSE bugs, you can close the entry by yourself, too.