Bug 925479

Summary: failure of loading in-kernel X.509 certificate
Product: [openSUSE] openSUSE Distribution Reporter: Heiko Rommel <heiko.rommel>
Component: KernelAssignee: Joey Lee <jlee>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: a.celli.casa, arvidjaar, forgotten_0Pg55wsCfH, forgotten_26iUOLiaH7, forgotten_7Vd19u3Vod, forgotten_ioUMYbuYMK, forgotten_ivgpbDCmcy, forgotten_QhwfAkIp0T, forgotten_TBwcE00DQs, goofi333, jlee, jsegitz, lnussel, meissner, mmarek, tiwai
Version: 13.2   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Heiko Rommel 2015-04-01 15:18:51 UTC
After applying the pending release candidate of the kernel

  openSUSE Maintenance ID: 3606
  openSUSE Review ID: 292232
  Repository URL: http://download.opensuse.org/repositories/openSUSE:/Maintenance:/3606/openSUSE_13.2_Update

I get the following message painted in red on my primary console:

[1.708945] Problem loading in-kernel X.509 certificate (-19)

However, I could not observe any run-time trouble (tested secure-boot with EFI, chained booting and direct booting with grub2). So it might be just a pretty-print issue - however a very visible one ;)

It seems to be a regression of the release candidate:

# uname -a; journalctl -b | grep X.509

  Linux f127 3.16.6-2-desktop #1 SMP PREEMPT Mon Oct 20 13:47:22 UTC 2014 (feb42ea) x86_64 x86_64 x86_64 GNU/Linux
  Apr 01 09:57:24 linux-dcfu kernel: Loading compiled-in X.509 certificates

  Linux f127 3.16.7-7-desktop #1 SMP PREEMPT Wed Dec 17 18:00:44 UTC 2014 (762f27a) x86_64 x86_64 x86_64 GNU/Linux
  Apr 01 10:48:07 linux-dcfu kernel: Loading compiled-in X.509 certificates

  Linux f127 3.16.7-13-desktop #1 SMP PREEMPT Wed Mar 18 17:31:15 UTC 2015 (ba2afab) x86_64 x86_64 x86_64 GNU/Linux
  Apr 01 10:53:33 linux-dcfu kernel: Loading compiled-in X.509 certificates
  Apr 01 10:53:33 linux-dcfu kernel: Problem loading in-kernel X.509 certificate (-19)
Comment 2 Marcus Meissner 2015-04-02 08:44:54 UTC
there are some candidates:

commit b0191b238e32268bb3b48b43928c8af3cf509eea
Author: Lee, Chun-Yi <jlee@suse.com>
Date:   Sat Jan 31 00:09:27 2015 +0800

    KEYS: close race between key lookup and freeing (bnc#912202).


commit 8a264a3b8772215f544b1d362967927515249279
Author: Michal Marek <mmarek@suse.cz>
Date:   Tue Dec 17 21:14:03 2013 +0100

    Port module signing changes from SLE11-SP3 (fate#314508)
Comment 4 Marcus Meissner 2015-04-07 11:20:01 UTC
(this is in the  13.2 kernel update we are currently qaing ... michal?)
Comment 5 Takashi Iwai 2015-04-07 13:19:47 UTC
The recent commit taken from SLE11-SP3 tries to convert *.crt files to *.x509, and avoid auto-generating the key.  But openSUSE has no keys set up:

[  393s] + for f in '/home/abuild/rpmbuild/SOURCES/*.crt'
[  393s] + test -e /home/abuild/rpmbuild/SOURCES/_projectcert.crt
[  393s] + out=_projectcert.crt
[  393s] + out=_projectcert.x509
[  393s] + openssl x509 -inform PEM -in /home/abuild/rpmbuild/SOURCES/_projectcert.crt -outform DER -out _projectcert.x509
[  394s] + touch x509.genkey
....

So it takes only the project default *.crt, no UEFI certificate.
Comment 6 Heiko Rommel 2015-04-08 07:23:55 UTC
(In reply to Takashi Iwai from comment #5)

> So it takes only the project default *.crt, no UEFI certificate.

What is the expected run-time effect of that ?

I am asking since - according to our test procedure for secure booting - secure-boot was reported as still working.
Comment 7 Takashi Iwai 2015-04-08 16:19:35 UTC
(In reply to Heiko Rommel from comment #6)
> (In reply to Takashi Iwai from comment #5)
> 
> > So it takes only the project default *.crt, no UEFI certificate.
> 
> What is the expected run-time effect of that ?
> 
> I am asking since - according to our test procedure for secure booting -
> secure-boot was reported as still working.

My wild guess is that it's just because you imported the same signature at the very first installation.  Or I might miss something obvious...
Comment 8 Takashi Iwai 2015-04-09 15:51:27 UTC
Joey must have a better clue.  Joey, could you check it?
Comment 9 Joey Lee 2015-04-10 03:43:32 UTC
The error -19 is ENODEV. It caused by asymmetric key type didn't build in to openSUSE kernel, but system keyring codes in kernel found a embedded key and try to using asymmetric key type to parse it, so no key type fail!

This issue exposed a couple of things need to think:

 - First: Looks SYSTEM_TRUSTED_KEYRING kernel config should select ASYMMETRIC_KEY_TYPE. I will send patch to upstream to add the changes in init/Kconfig

 - Second: the SYSTEM_TRUSTED_KEYRING a.k.a. system kerying  is used by: MODULE_SIG, IMA, PKCS7(new), KEXEC_BZIMAGE_VERIFY_SIG(new),
   none of above functions enabled on openSUSE 13.2. So, we should think may disable SYSTEM_TRUSTED_KEYRING from openSUSE kernel.

 - Third: Per dmesg, looks load_system_certificate_list() codes found a embedded key in kernel binary then tried to parser it. If we don't need kernel module sign or IMA. Maybe we can remove the embedded key in kernel.


I will disable SYSTEM_TRUSTED_KEYRING in openSUSE 13.2 and openSUSE Factory until we enable IMA or kexec binary verification in future openSUSE version.
Comment 10 Forgotten User QhwfAkIp0T 2015-04-13 15:58:13 UTC
Experiencing the same bug with the recent kernel-desktop-3.16.7-13.2 update.
Comment 11 Forgotten User ivgpbDCmcy 2015-04-13 18:31:59 UTC
(In reply to e p from comment #10)
> Experiencing the same bug with the recent kernel-desktop-3.16.7-13.2 update.

Me too. (Mainboard ASRock 960GM/U3S3 FX)
Comment 12 Takashi Iwai 2015-04-13 18:48:21 UTC
*** Bug 926972 has been marked as a duplicate of this bug. ***
Comment 13 Takashi Iwai 2015-04-13 18:51:04 UTC
(In reply to Philipp - from comment #11)
> (In reply to e p from comment #10)
> > Experiencing the same bug with the recent kernel-desktop-3.16.7-13.2 update.
> 
> Me too. (Mainboard ASRock 960GM/U3S3 FX)

This message must appear on every machine no which variant is.  And, it's harmless, after all.  So, don't worry.
Comment 14 Takashi Iwai 2015-04-13 18:52:06 UTC
(In reply to Joey Lee from comment #9)
> I will disable SYSTEM_TRUSTED_KEYRING in openSUSE 13.2 and openSUSE Factory
> until we enable IMA or kexec binary verification in future openSUSE version.

Yeah, I agree with it.  The current situation with an error message just confuses users.
Comment 15 Forgotten User 26iUOLiaH7 2015-04-13 19:19:05 UTC
As a result of the issue my nvidia driver will not load anymore. They seem related to these: 

https://ask.fedoraproject.org/en/question/29067/problem-loading-in-kernel-x509-certificate-129/

http://superuser.com/questions/788401/adding-x-509-certificate-to-uefi-secure-boot-database 

So more annoying than just a console message during boot. . .
Comment 16 Forgotten User TBwcE00DQs 2015-04-13 20:51:55 UTC
The same with nvidia - X fails to start. Great to have this on stable branch ;)
Any workaround?
Comment 17 Forgotten User 0Pg55wsCfH 2015-04-14 01:48:03 UTC
X failing to start with nvidia drivers after updating the kernel has nothing to do with the problem with the x.509 certificate.

It's a different bug, a big one I think. After updating the kernel the previously installed binary from nvidia no longer works, because it needs some on-machine compilation process to link module against the new kernel and install it properly.

Just re-install the nvidia rpm packages and it should work again.

Opensuse guys.. how do you expect a normal user to react when after appliying an update and rebooted their systems as indicated, they stand up in front of a text terminal? They're not supposed to know what's happening, and how to solve it.

Quite frustrating don't you think so?

Updating the kernel should have in consideration if the user has nvidia drivers installed, or broadcom wl, or any other drivers which can lead to such a situation, and do an automatic re-installation of those packages, or at last, alert the user of what's gonna happen.

Best regards.
Comment 18 Joey Lee 2015-04-14 02:16:20 UTC
(In reply to stephan stephan from comment #15)
> As a result of the issue my nvidia driver will not load anymore. They seem
> related to these: 
> 
> https://ask.fedoraproject.org/en/question/29067/problem-loading-in-kernel-
> x509-certificate-129/
> 
> http://superuser.com/questions/788401/adding-x-509-certificate-to-uefi-
> secure-boot-database 
> 
> So more annoying than just a console message during boot. . .

The -129 error is EKEYREJECTED, it's not the same with the -29 on this issue. 

The kernel module signature verification function disabled by openSUSE community, so there should not have such error on openSUSE.

Before v3.13 kernel, the -129 error may caused by certificate date check, that's already removed by 124df9260 patch since v3.13.

In driver verify process, the -129 error means the hash value in digest doesn't match with the hash that we calculated in verification process. That may caused by the certificate doesn't match with the sign key of driver. 

Simply said, 2 cases. Both of them should not happened on openSUSE 13.2:

 - If you see "EFI: Problem loading in-kernel X.509 certificate (-129)"
   That should not happened after v3.13 kerel, please try newer kernel. If you sill found -129 when loading in-kernel certificate, please file bug to kernel upstream.

 - If you see -129 when loading driver, then please check does appropriate certificate already enrolled to db or mok.
Comment 19 Joey Lee 2015-04-14 02:19:00 UTC
(In reply to Joey Lee from comment #18)
[snip]
> The -129 error is EKEYREJECTED, it's not the same with the -29 on this issue
                                                             ^^^^
                                                             -19, sorry for typo
Comment 20 Andrei Borzenkov 2015-04-14 05:58:47 UTC
(In reply to Andrey Polunkin from comment #16)
> The same with nvidia - X fails to start.

https://bugzilla.opensuse.org/show_bug.cgi?id=927018

> Any workaround?

Rebuild driver for new kernel or boot previous version. If you do not know how to rebuild - remove RPM and reinstall.
Comment 21 Joey Lee 2015-04-14 06:39:01 UTC
Disable CONFIG_SYSTEM_TRUSTED_KEYRING in kernel config files for openSUSE-13.2, master (Factory) and stable (Tumbleweed):

openSUSE 13.2:

commit 55df6c23cae67820332ec0854288bf0ae5a3f51c
Author: Lee, Chun-Yi <jlee@suse.com>
Date:   Tue Apr 14 12:18:10 2015 +0800

    Update config files. (boo#925479)
    
    Do not set CONFIG_SYSTEM_TRUSTED_KEYRING until we need it in future
    openSUSE version:
    e.g. MODULE_SIG, IMA, PKCS7(new), KEXEC_BZIMAGE_VERIFY_SIG(new)


stable (Tumbleweed):

commit 74c332ba11409dc319e7ed43fbf41416eabe8534
Author: Lee, Chun-Yi <jlee@suse.com>
Date:   Tue Apr 14 12:39:50 2015 +0800

    Update config files. (boo#925479)
    
    Do not set CONFIG_SYSTEM_TRUSTED_KEYRING until we need it in future
    openSUSE version:
    e.g. MODULE_SIG, IMA, PKCS7(new), KEXEC_BZIMAGE_VERIFY_SIG(new)


master (Factory):

commit 5c4d917abf1a10aa0303223a027597bca837e4af
Author: Lee, Chun-Yi <jlee@suse.com>
Date:   Tue Apr 14 14:32:12 2015 +0800

    Update config files. (boo#925479)
    
    Do not set CONFIG_SYSTEM_TRUSTED_KEYRING until we need it in future
    openSUSE version:
    e.g. MODULE_SIG, IMA, PKCS7(new), KEXEC_BZIMAGE_VERIFY_SIG(new)
Comment 22 Heiko Rommel 2015-04-14 07:13:29 UTC
(In reply to Andrei Borzenkov from comment #20)
> (In reply to Andrey Polunkin from comment #16)
> > The same with nvidia - X fails to start.
> 
> https://bugzilla.opensuse.org/show_bug.cgi?id=927018
> 
> > Any workaround?
> 
> Rebuild driver for new kernel or boot previous version. If you do not know
> how to rebuild - remove RPM and reinstall.

Please note that the NVIDIA (3rd party) drivers suffer from an unrelated problem which might be impacting your case - see https://bugzilla.suse.com/show_bug.cgi?id=925437
Comment 23 Joey Lee 2015-04-14 07:25:01 UTC
Set this issue to FIXED.

CONFIG_SYSTEM_TRUSTED_KEYRING disabled in kernel config files for openSUSE-13.2, master (Factory) and stable (Tumbleweed). Please feel free to reopen this issue if you still see problem in new kernel.
Comment 24 Forgotten User 7Vd19u3Vod 2015-04-14 11:02:11 UTC
I got the same message after ugrading to the new kernel version yesterday (13th April) - 3.16.7-17-desktop
Problem loading in-kernel X.509 certificate (-19)

Furthermore I got a x windows problem after the kernel upgrade - NVIDIA graphic drivers not starting properly (this never happened before after kernel upgrades)
This problem could be solved by refreshing the NVIDIA driver installation via YaST.
Comment 25 Andrea Celli 2015-04-14 12:05:10 UTC
Thanks Alexander,
   but i'm not a smart user :-(
How can i "Refresh the NVIDIA driver installation via YaST"?

I started Yast -> Software management -> Search Nvidia and i found
i    X11-videonvidia-G02
i    xf86-video-nouveau
i    xf86-video-nv

Actually i was using the first one.

As possible actions i find:
toggle
Install
delete
update
Taboo
lock

Do i have to delete driver and install it again?

By the way, can i remove NV and Nouveau drivers?

bye, andrea
Comment 26 Forgotten User 7Vd19u3Vod 2015-04-14 12:41:36 UTC
(In reply to Andrea Celli from comment #25)
> Thanks Alexander,
>    but i'm not a smart user :-(
> How can i "Refresh the NVIDIA driver installation via YaST"?
> 
> I started Yast -> Software management -> Search Nvidia and i found
> i    X11-videonvidia-G02
> i    xf86-video-nouveau
> i    xf86-video-nv
> 
> Actually i was using the first one.
> 
> As possible actions i find:
> toggle
> Install
> delete
> update
> Taboo
> lock
> 
> Do i have to delete driver and install it again?
> 
> By the way, can i remove NV and Nouveau drivers?
> 
> bye, andrea

Hi Andrea,

first of all I was lucky that some low resolution X windows system popped up instead of NVIDIA. Therefore I was still able to use the graphical YaST UI.
Furthermore you seem to have an older NVIDIA version than I have.

I selected in YaST to display the software source NVIDIA and then selected "update" for all as installed listed rpms. 
This should also be possible with the text version of YaST. 

If this doesn't work, delete the NVIDIA and check whether it then starts with NV or Noveau.

Best regards,

Alexander
Comment 27 Marcus Meissner 2015-04-14 15:28:37 UTC
JFY, i removed the current 13.2 kernel update from the server.

it will be replaced with a newer one with this issue not appearing.
Comment 28 Forgotten User 7Vd19u3Vod 2015-04-14 17:23:40 UTC
(In reply to Alexander König from comment #24)
> I got the same message after ugrading to the new kernel version yesterday
> (13th April) - 3.16.7-17-desktop
> Problem loading in-kernel X.509 certificate (-19)
> 
> Furthermore I got a x windows problem after the kernel upgrade - NVIDIA
> graphic drivers not starting properly (this never happened before after
> kernel upgrades)
> This problem could be solved by refreshing the NVIDIA driver installation
> via YaST.

Typo error: not 3.16.7-17-desktop but 3.16.7-13-desktop
Comment 29 Forgotten User 7Vd19u3Vod 2015-04-25 13:16:02 UTC
Update to 3.16.7-21-desktop on 13.2 required refreshment of NVIDIA driver - I had the 3.16.7-13-desktop kernel installed and my NVIDIA module refreshed for that