Bug 1100427

Summary: Unable to install linux kernel via dud
Product: [openSUSE] openSUSE Tumbleweed Reporter: Michel Normand <normand>
Component: libzyppAssignee: E-mail List <zypp-maintainers>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: jreidinger, normand, snwint, zypp-maintainers
Version: Current   
Target Milestone: ---   
Hardware: PowerPC   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: serial0.txt
logs_from_installation_system-y2logs.tar.bz2
unknown_gnupg_key.png
not_applicable_dud_linuxrc.log
linuxrc_with_mksusecd_initrd_changed_parameter.log
serial0.txt
logs_from_installation_system-y2logs.tar.bz2
await_install-y2logs.tar.bz2
logs_from_installation_system-y2logs.tar.bz2
gpg key and signed CHECKSUMS file

Description Michel Normand 2018-07-06 14:45:57 UTC
Created attachment 776377 [details]
serial0.txt

Unable to install a new linux kernel via dud

* I locally built a linux kernel, then call mkdud and mksusecd to create a dud file and a new iso from original Tumbleweed ppc64le iso.

* The Tumbleweed new iso successfully installed.
 
* I confirmed the new kernel was used during install process,
  as per serial0.txt extract (captured before User Selection in Yast UI)
  ===
  uname -a:
  Linux install 4.17.3-mnotrial4.0-default
  ===

* BUT Yast installed the original kernel and not the new one passed via dud.

* I was using:
  mkdud v1.35
  mksusecd v1.62  (with correction of bug#1100236)

* During Yast menu, there was an "Unknown GnuPG Key" warning, that I decided to accept to be able to complete the install.

* at the end of the install the installed kernel is not the one from dud.
  as per serial0.txt
  ===  
  Welcome to openSUSE Tumbleweed 20180704 - Kernel 4.17.3-1-default (hvc0).
  enp0s2: 10.0.2.15 fec0::2dd8:41ab:b7d5:ecdf
  linux-8y9g login:
  ===
Comment 1 Michel Normand 2018-07-06 15:11:38 UTC
Created attachment 776378 [details]
logs_from_installation_system-y2logs.tar.bz2

the attached logs_from_installation_system-y2logs.tar.bz2 was captured after the install before the reboot.

There are at least two problems:
* I have in y2start.log the dud parameter, but no trace in y2log of the loading of "kernel-default-4.17.3-mnotrial4.0" coming from dud.
* There is not traces of "Unknown GnuPG Key" warning previously reporting via Yast GUI (and that I ignored typing Y to related prompt).
===
$grep -Hnr mnotrial4 ./
Xorg.0.log:7:[    62.044] Current Operating System: Linux install 4.17.3-mnotrial4.0-default #1 SMP mar. juin 26 08:45:20 CEST 2018 (e8dc1b5) ppc64le
Xorg.0.log:8:[    62.044] Kernel command line: BOOT_IMAGE=/boot/ppc64le/linux Y2DEBUG=1 gpgcheck=off dud=ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud insecure=1
boot.msg:7:<5>Linux version 4.17.3-mnotrial4.0-default (geeko@buildhost) (gcc version 8.1.1 20180523 [gcc-8-branch revision 260570] (SUSE Linux)) #1 SMP mar. juin 26 08:45:20 CEST 2018 (e8dc1b5)
boot.msg:58:<5>Kernel command line: BOOT_IMAGE=/boot/ppc64le/linux Y2DEBUG=1 gpgcheck=off dud=ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud insecure=1
boot.msg:482:<6>usb usb1: Manufacturer: Linux 4.17.3-mnotrial4.0-default xhci-hcd
boot.msg:493:<6>usb usb2: Manufacturer: Linux 4.17.3-mnotrial4.0-default xhci-hcd
linuxrc.log:19:13:55:04 <4>: options[dud] = "ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud"
linuxrc.log:127:13:55:25 <2>: dud url: ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud
linuxrc.log:128:13:55:25 <4>: url = ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud
linuxrc.log:129:13:55:25 <1>: Reading driver update: ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud
linuxrc.log:132:13:55:25 <2>: url_setup_device: ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud?device=enp0s2
linuxrc.log:155:13:55:42 <4>: url = ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud
linuxrc.log:156:13:55:42 <2>: loading ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud -> /download/file_0000
linuxrc.log:196:13:55:46 <1>: ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud: adding to installation system
linuxrc.log:197:13:55:46 <2>: ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud -> /download/dud_0000: converting dud to squashfs
linuxrc.log:227:13:55:48 <1>:   pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud
YaST2/y2start.log:48:	|-- declare -x dud="ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud"
messages:7:2018-07-06T13:55:55.372472+00:00 install kernel: Linux version 4.17.3-mnotrial4.0-default (geeko@buildhost) (gcc version 8.1.1 20180523 [gcc-8-branch revision 260570] (SUSE Linux)) #1 SMP mar. juin 26 08:45:20 CEST 2018 (e8dc1b5)
messages:58:2018-07-06T13:55:55.372547+00:00 install kernel: Kernel command line: BOOT_IMAGE=/boot/ppc64le/linux Y2DEBUG=1 gpgcheck=off dud=ftp://sf1.test.toulouse-stg.fr.ibm.com/pub/linux/opensuse/factory/ppc64le/iso/mnotrial4.dud insecure=1
messages:481:2018-07-06T13:55:55.373259+00:00 install kernel: usb usb1: Manufacturer: Linux 4.17.3-mnotrial4.0-default xhci-hcd
messages:492:2018-07-06T13:55:55.373275+00:00 install kernel: usb usb2: Manufacturer: Linux 4.17.3-mnotrial4.0-default xhci-hcd
messages:701:2018-07-06T14:03:18.687035+00:00 install os-prober: modprobe: FATAL: Module fuse not found in directory /lib/modules/4.17.3-mnotrial4.0-default
messages:708:2018-07-06T14:03:43.136492+00:00 install os-prober: modprobe: FATAL: Module fuse not found in directory /lib/modules/4.17.3-mnotrial4.0-default
===
Comment 2 Michel Normand 2018-07-06 15:29:21 UTC
Initial mkdud traces:
===
+ mkdud --create mnotrial4.dud --dist sles12 --install repo ./k
===  Update #1  ===
  [SLES 12]
    Name:
      Update 3322c9e3-a491-4928-aa12-dc58839c87bd
    ID:
      3322c9e3-a491-4928-aa12-dc58839c87bd
    Installation System:
      /kernel-default-4.17.3-mnotrial4.0.ppc64le.rpm
===
Comment 3 Michel Normand 2018-07-06 15:35:00 UTC
mksusecd traces:
===
+ sudo mksusecd --verbose --verbose --create mnotrial4.iso --rebuild-initrd --initrd mnotrial4.dud --kernel ./k/kernel-default-4.17.3-mnotrial4.0.ppc64le.rpm ./k/kernel-firmware-20180606-1.1.noarch.rpm -- /home/michel/iso/openSUSE-Tumbleweed-DVD-ppc64le-Build502.1-Media.iso
mksquashfs has '-comp': 1
Repositories:
  openSUSE [20180704-0]
assuming repo-md sources
added linuxrc option InstSys="disk:/boot/ppc64le/root"
transient signing key created, keyid = 75900CC5FA5E511A
signing key added to initrd
kernel version: 4.17.3-1-default --> 4.17.3-mnotrial4.0-default
re-signing '/CHECKSUMS'
CHRP bootable (ppc64le)
running:
/usr/bin/mkisofs -U -l -r -pad -input-charset utf8 -o 'mnotrial4.iso' -V 'openSUSE-Tumbleweed-DVD-ppc64le5' -A 'openSUSE-Tumbleweed-DVD-ppc64le-Build502.1-Media' -p 'KIWI - http://opensuse.github.com/kiwi' -publisher 'SUSE LINUX GmbH' -J -f -joliet-long -exclude-list '/tmp/mksusecd.T2qegj7m/exclude' -graft-points -path-list '/tmp/mksusecd.T2qegj7m/filelist'
^Mbuilding: 100%
running:
isohybrid  --no-code --no-chs --type 0x96 'mnotrial4.iso'
isohybrid: Warning: more than 1024 cylinders: 3621
isohybrid: Not all BIOSes will be able to boot this device
calculating sha256...
===
Comment 4 Michel Normand 2018-07-06 15:37:21 UTC
Steffen do you have a suggestion on how to continue investigation ?
Comment 5 Michel Normand 2018-07-06 16:03:33 UTC
Created attachment 776385 [details]
unknown_gnupg_key.png

this is the warning reported by Yast during the install,
Warning I bypassed typing the Yes key,
that allow to continue the install process up to the end.

BUT not with the mnotrial4 kernel part of dud file.
Comment 6 Steffen Winterfeldt 2018-07-09 07:42:49 UTC
Did you really do 'mkdud --dist sles12'? This would create a driver update
for sle12. For Tumbleweed use '--dist tw' (you also use both if you like).
Comment 7 Michel Normand 2018-07-09 10:16:57 UTC
(In reply to Steffen Winterfeldt from comment #6)
> Did you really do 'mkdud --dist sles12'? This would create a driver update
> for sle12. For Tumbleweed use '--dist tw' (you also use both if you like).

I just did a trial with --dist tw --obs-keys, but trial of install failed now with linuxrc: "no applicable driver updates found", I will need to retrieve linurc logs ...

I previously used the same mksusecd parameters and it traces same as in comment#3
Comment 8 Michel Normand 2018-07-09 12:53:29 UTC
Created attachment 776460 [details]
not_applicable_dud_linuxrc.log

capture the linuxrc.log with "No applicable driver updates found"
There is no obvious cause of the error reported in the log.
=== extract of linuxrc.log
11:33:34 <2>: dud: ftp:/linux/suse/ppc64le-tw (duplicate of 0, skipped)
11:33:34 <2>: dud 0:
11:33:34 <2>:   Update 12e1688a-3f3e-4e62-a347-16ae54e59361                                                            
11:33:34 <1>: Driver Update: Update 12e1688a-3f3e-4e62-a347-16ae54e59361                                               
11:33:34 <2>: No applicable driver updates found.
===

How to continue investigation ?

The previously used mkdud was version 1.35, and its parameters and stdout were:
===
+ mkdud --create mnotrial4.dud --dist tw --install repo --obs-keys ./k
===  Update #1  ===
  [openSUSE Tumbleweed]
    Name:
      Update 12e1688a-3f3e-4e62-a347-16ae54e59361
    ID:
      12e1688a-3f3e-4e62-a347-16ae54e59361
    Installation System:
      /kernel-default-4.17.3-mnotrial4.0.ppc64le.rpm
===
Comment 9 Steffen Winterfeldt 2018-07-09 13:33:26 UTC
11:32:57 <2>: dud 0: /:/linux/suse/ppc64le-tw

This "/:" looks suspicious. You're sure the driver update has been added to
the initrd with the correct path?
Comment 10 Michel Normand 2018-07-09 14:04:46 UTC
(In reply to Steffen Winterfeldt from comment #9)
> 11:32:57 <2>: dud 0: /:/linux/suse/ppc64le-tw
> 
> This "/:" looks suspicious. You're sure the driver update has been added to
> the initrd with the correct path?

I added the dud with syntax specified in comment#3 : 
mksusecd ... --initrd mnotrial4.dud

as suggested in wiki https://lizards.opensuse.org/2017/03/16/fun-things-to-do-with-driver-updates-2/ : 
mksuscd ... --initrd foo.dud
Comment 11 Michel Normand 2018-07-09 14:10:10 UTC
I am just trying with --initrd ./mnotrial4.dud
Comment 12 Michel Normand 2018-07-09 14:50:49 UTC
Created attachment 776473 [details]
linuxrc_with_mksusecd_initrd_changed_parameter.log

(In reply to Michel Normand from comment #11)
> I am just trying with --initrd ./mnotrial4.dud

the linuxrc.log related to new mnotrial4.iso with changed --initrd parameter has the same dud 0: /:/linux/suse/ppc64le-tw as identified in comment#9 by Steffen.
===
$grep dud linuxrc.log
14:23:30 <2>: dud 0: /:/linux/suse/ppc64le-tw
14:23:44 <2>: dud url: disk:/?device=*usb*&all=1&quiet=1
14:23:44 <2>: dud url: disk:/?device=*label/OEMDRV&quiet=1
14:23:44 <2>: dud 0:
===
Comment 13 Michel Normand 2018-07-09 16:13:54 UTC
(In reply to Steffen Winterfeldt from comment #9)
> 11:32:57 <2>: dud 0: /:/linux/suse/ppc64le-tw
> 
> This "/:" looks suspicious. You're sure the driver update has been added to
> the initrd with the correct path?

If manually copy and unpack the initrd from new mnotrial4.iso, then I am able to find the files in /linux/suse/ppc64le-tw:
===
$cp initrd /tmp/tmp
$cd /tmp/tmp
$xz -dc initrd | cpio -id
...
$find linux/suse/ppc64le-tw/
linux/suse/ppc64le-tw/
linux/suse/ppc64le-tw/inst-sys
linux/suse/ppc64le-tw/inst-sys/kernel-default-4.17.3-mnotrial4.0.ppc64le.rpm
linux/suse/ppc64le-tw/dud.config
===
$ls -ltrah linux/suse/ppc64le-tw/*
-rw-r--r-- 1 normand ibm  127 juil.  9 18:04 linux/suse/ppc64le-tw/dud.config

linux/suse/ppc64le-tw/inst-sys:
total 55M
drwxr-xr-x 2 normand ibm 4,0K juil.  9 18:04 .
-rw-r--r-- 1 normand ibm  55M juil.  9 18:04 kernel-default-4.17.3-mnotrial4.0.ppc64le.rpm
drwxr-xr-x 3 normand ibm 4,0K juil.  9 18:04 ..
===
$cat linux/suse/ppc64le-tw/dud.config
# created by mkdud 1.35
UpdateID:       12e1688a-3f3e-4e62-a347-16ae54e59361
UpdateName:     Update 12e1688a-3f3e-4e62-a347-16ae54e59361
===
Comment 14 Steffen Winterfeldt 2018-07-10 08:41:59 UTC
Ok, everything seems to be fine. The message in comment 7 is a red herring. :-)

You passed the driver update two times: once directly in the initrd and another
time via ftp. The message refers to the second try via ftp.

So, things are fine and he kernel should be installed. If you're unsure you
can enter the package manager at the summary screen shown right before you'd
start the installation. Check the kernel package - it should come from a
'DriverUpdate' repository.
Comment 15 Michel Normand 2018-07-10 10:04:26 UTC
Created attachment 776546 [details]
serial0.txt

as suggested I removed the DUD=... boot parameter, and so do not have anymore the "no applicable driver updates found" message of comment#7

BUT in serial console, the new kernel used during install (4.17.4-mnotrial4.0-default) IS NOT the installed  kernel  (4.17.4-1-default)
===
$grep -n -A5 -B2 4.17.4 ~/Desktop/serial0.txt 
775-yast2-ycp-ui-bindings [4.0.0-1.2.ppc64le] < yast2
776-uname -a:
777:Linux install 4.17.4-mnotrial4.0-default #1 SMP mar. juil. 3 13:31:47 CEST 2018 (882c99b) ppc64le ppc64le ppc64le GNU/Linux
778-ZE144-0-
779-N06xI-0-
780-IKG0B-0-
781-KdUfq-0-
782-T_TZ1-0-
--
Linux ppc64le
861-#1 SMP Tue Jul 3
862:Welcome to openSUSE Tumbleweed 20180707 - Kernel 4.17.4-1-default (hvc0).
863-
864-enp0s2: 10.0.2.15 fec0::90b8:c364:53c5:6c7f
865-
866-
867-linux-p7jt login: 
===
Comment 16 Steffen Winterfeldt 2018-07-10 11:19:04 UTC
Then please do the check I outlined in comment 14.
Comment 17 Michel Normand 2018-07-10 12:05:36 UTC
Created attachment 776562 [details]
logs_from_installation_system-y2logs.tar.bz2

(In reply to Steffen Winterfeldt from comment #16)
> Then please do the check I outlined in comment 14.

I will do later.
But anyway the y2log extracted from attached logs_from_installation_system-y2logs.tar.bz2 is reporting install of old kernel and not new one:
===
[zypp] PackageProvider.cc(providePackage):296 provide Package (1208)kernel-default-4.17.4-1.3.ppc64le(openSUSE-20180709-0)
...
[zypp::exec++] ExternalProgram.cc(start_program):252 Executing 'rpm' '--root' '/mnt' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/mnt/var/cache/zypp/packages/openSUSE-20180709-0/ppc64le/kernel-default-4.17.4-1.3.ppc64le.rpm'
===
Comment 18 Steffen Winterfeldt 2018-07-10 13:06:50 UTC
Found it:

> mkdud ... --obs-keys ./k

is wrong; it must be

mkdud ... --obs-keys ./k/*
Comment 19 Michel Normand 2018-07-10 16:04:17 UTC
modify as suggested the mkdud parameters: mkdud ... --obs-keys ./k/*
now yast has a "DriverUpdate0" repo, 
but ultimately failed when trying to download kernel-default:
"Package kernel-default is broken, integrity check has failed"
I thought the --obs-keys was supposed to cover such problem.
I will need to read yast log
Comment 20 Michel Normand 2018-07-11 08:51:13 UTC
Created attachment 776672 [details]
await_install-y2logs.tar.bz2

The y2log extracted from attached bz2 file, confirms failure of kernel-default not signed.

Is there an option to pass as boot parameter to ignore such warning ?

=== extract y2log:
2018-07-10 12:40:44 <1> install(2743) [zypp:fetcher] Fetcher.cc(validate):366 Checking job [/update/000/repo/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm] (0 checkers )
2018-07-10 12:40:44 <1> install(2743) [zypp] PathInfo.cc(hardlinkCopy):863 hardlinkCopy /update/000/repo/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm -> /mnt/var/cache/zypp/packages/DriverUpdate0/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm => copy
2018-07-10 12:40:44 <1> install(2743) [zypp::exec++] ExternalProgram.cc(start_program):252 Executing '/bin/cp' '--remove-destination' '--' '/update/000/repo/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm' '/mnt/var/cache/zypp/packages/DriverUpdate0/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm'
2018-07-10 12:40:44 <1> install(2743) [zypp::exec++] ExternalProgram.cc(start_program):415 pid 15783 launched
2018-07-10 12:40:44 <1> install(2743) [zypp::exec++] ExternalProgram.cc(checkStatus):516 Pid 15783 successfully completed
2018-07-10 12:40:44 <1> install(2743) [zypp] PathInfo.cc(copy):800 copy /update/000/repo/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm -> /mnt/var/cache/zypp/packages/DriverUpdate0/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm 
2018-07-10 12:40:44 <1> install(2743) [zypp++] MediaSetAccess.cc(releaseFile):86 Going to release file ./kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm from media number 1
2018-07-10 12:40:44 <1> install(2743) [Progress++] ProgressData.cc(report):88 {#1938|}END
2018-07-10 12:40:44 <1> install(2743) [zypp] RepoProvideFile.cc(provideFile):320 provideFile at /mnt/var/cache/zypp/packages/DriverUpdate0/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm
2018-07-10 12:40:44 <2> install(2743) [zypp] RpmDb.cc(doCheckPackageSig):1607 /mnt/var/cache/zypp/packages/DriverUpdate0/kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm (1 -> [6-File is unsigned])
2018-07-10 12:40:44 <2> install(2743) [zypp] RpmDb.cc(doCheckPackageSig):1608 kernel-default-4.17.4-mnotrial5.0.ppc64le.rpm:
2018-07-10 12:40:44 <2> install(2743) [zypp] RpmDb.cc(doCheckPackageSig):1608     Header SHA1 digest: OK
2018-07-10 12:40:44 <2> install(2743) [zypp] RpmDb.cc(doCheckPackageSig):1608     Header SHA256 digest: OK
2018-07-10 12:40:44 <2> install(2743) [zypp] RpmDb.cc(doCheckPackageSig):1608     Payload SHA256 digest: OK
2018-07-10 12:40:44 <2> install(2743) [zypp] RpmDb.cc(doCheckPackageSig):1608     MD5 digest: OK
2018-07-10 12:40:44 <0> install(2743) [Pkg] Callbacks.YCP.cc(createCallback):198 Callback PkgGpgCheck is empty
2018-07-10 12:40:44 <0> install(2743) [Pkg] Callbacks.YCP.cc(evaluate):240 Evaluating callback (registered funciton: ruby_reference)
2018-07-10 12:40:44 <1> install(2743) [Ruby] modules/PackageCallbacks.rb:201 DoneProvide: 3, kernel-default-4.17.4-mnotrial5.0.ppc64le (DriverUpdate0): Signature verification failed [6-File is unsigned]
    Header SHA1 digest: OK
    Header SHA256 digest: OK
    Payload SHA256 digest: OK
    MD5 digest: OK
    Package is not signed!
, kernel-default
===
Comment 21 Steffen Winterfeldt 2018-07-11 09:01:42 UTC
> Is there an option to pass as boot parameter to ignore such warning ?

No. But you can modify /etc/zypp/zypp.conf to turn checks off.

> Package is not signed!

Or sign the package.
Comment 22 Michel Normand 2018-07-11 15:07:08 UTC
With the new kernel signed with a gnupg key, and related public key added as source of mkdud, then yast is able to install the new kernel, so no more failure of comment#19.

BUT, still have "unknown GnuPG key" warning message previously reported in comment#5

From where is coming the failing CHECKSUMS file ? is it generated/modified by one of two scripts mkdud or mksusecd ?
Comment 23 Josef Reidinger 2018-08-16 09:15:25 UTC
steffen please see comment#22 if you cna answer. Thanks
Comment 24 Michel Normand 2018-08-16 13:58:49 UTC
Created attachment 779940 [details]
logs_from_installation_system-y2logs.tar.bz2

I just retried with last TW and captured the logs_from_installation_system-y2logs.tar.bz2.  From this archive file the y2log-3.gz has following lines that confirm the GnuPG was initially trusted by SignatureCheckCallbacks.rb but then considered as unknown by zypp:KeyRing :(

===
=== extract y2log-3.gz
===
2018-08-16 08:25:46 <1> install(2777) [Ruby] modules/SignatureCheckCallbacks.rb:285 Trusted key has been added: F7B333B196C8009B / FA85E6362A1F940DD81D74A5F7B333B196C8009B (mksusecd Signing Key (transient key))
2018-08-16 08:25:47 <1> install(2777) [Ruby] modules/SignatureCheckCallbacks.rb:285 Trusted key has been added: F7B333B196C8009B / FA85E6362A1F940DD81D74A5F7B333B196C8009B (mksusecd Signing Key (transient key))
...
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(readSignatureKeyId):570 Determined key id [FA85E6362A1F940DD81D74A5F7B333B196C8009B] for signature /var/tmp/TmpDir.r4qmV1/CHECKSUMS.asc
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(publicKeyExists):351 No key [FA85E6362A1F940DD81D74A5F7B333B196C8009B] in keyring /var/tmp/zypp.WihHxm/zypp-trusted-krmczjiP
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153 Found keys: {
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153   [B88B2FD43DBDC284-53674dd4] [openSUSE Project Signing Key <opensuse@opensuse.org>] [expires: 2024-05-02]
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153   [F7B333B196C8009B-5b7567e6] [mksusecd Signing Key (transient key)] [does not expire]
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153 }
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(publicKeyExists):351 No key [FA85E6362A1F940DD81D74A5F7B333B196C8009B] in keyring /var/tmp/zypp.WihHxm/zypp-general-krU8RV2h
2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(verifyFileSignatureWorkflow):497 File [/mounts/mp_0000/CHECKSUMS] ( CHECKSUMS ) signed with unknown key [FA85E6362A1F940DD81D74A5F7B333B196C8009B]
...
2018-08-16 08:29:00 <0> install(2777) [Ruby] binary/Yast.cc(ycp_module_call_ycp_function):332 Append parameter `VBox (`Heading ("Unknown GnuPG Key"), `MarginBox (0.5, 0.5, `Label ("The file CHECKSUMS\nis digitally signed with the following unknown GnuPG key: \nID: FA85E6362A1F940DD81D74A5F7B333B196C8009B.\n\nThis means that a trust relationship to the creator of the file\ncannot be established. Using the file may put the integrity\nof your system at risk.\n\nUse it anyway?")), `Left (`MarginBox (0, 1.2, `CheckBox (`id (`dont_show_again), "Do Not Show This Message &Again", false))), `ButtonBox (`PushButton (`id (`yes), `opt (`okButton, `key_F10), "&Yes"), `PushButton (`id (`no), `opt (`default, `cancelButton, `key_F9), "&No")))
===
Comment 25 Steffen Winterfeldt 2018-08-16 15:24:56 UTC
> 2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153 Found keys: {
> 2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153   [B88B2FD43DBDC284-53674dd4] [openSUSE Project Signing Key <opensuse@opensuse.org>] [expires: 2024-05-02]
> 2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153   [F7B333B196C8009B-5b7567e6] [mksusecd Signing Key (transient key)] [does not expire]
> 2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(getData):153 }
> 2018-08-16 08:29:00 <1> install(2777) [zypp::KeyRing] KeyRing.cc(publicKeyExists):351 No key [FA85E6362A1F940DD81D74A5F7B333B196C8009B] in keyring /var/tmp/zypp.WihHxm/zypp-general-krU8RV2h

IOW: it doesn't find the key even though it's apparently part of the keyring.

Michael, any idea?
Comment 26 Michael Andres 2018-08-16 17:11:04 UTC
I'd need a copy of key:
> [F7B333B196C8009B-5b7567e6] [mksusecd Signing Key (transient key)] [does not expire]

Where do I find it?

While there is no problem with e.g. the openSUSE Project Signing Key  [B88B2FD43DBDC284-53674dd4], the above key is not found in any of the keyrings, though dumping the keyring shows it it. That's strange.
Comment 27 Michel Normand 2018-08-16 17:27:37 UTC
* I created the kernel-default-4.17.14-20180816.gdc49b43.ppc64le rpm and signed it with my own key. 
* Then added the GnuPG key in dud with this rpm.
* and embed the dud in iso before to try to install.

The mkdud log was:
```
+ mkdud --create opensuse_20180816.dud --dist tw --install repo --obs-keys ./k/kernel-default-4.17.14-20180816.gdc49b43.ppc64le.rpm ./k/RPM-GPG-KEY-michelmno
./k/kernel-default-4.17.14-20180816.gdc49b43.ppc64le.rpm: obs info missing, can't get sign key
===  Update #1  ===
  [openSUSE Tumbleweed (ppc64le)]
    Name:
      kernel-default-4.17.14-20180816.gdc49b43.ppc64le  Thu Aug 16 11:05:23 2018
    ID:
      00448a58-3be2-4eed-83bf-b31fa39bec58
    Packages:
      kernel-default-4.17.14-20180816.gdc49b43.ppc64le.rpm  (Thu Aug 16 11:05:23 2018)
      - install methods: repo (repo priority 50)
    Scripts:
      2 x update.pre, update.post2
    RPM Public Keys:
      gpg-pubkey-411ea9c9-5b45ced2.asc
      (Michel Normand <normand@linux.vnet.ibm.com> [expires: 2020-07-10])
```
Comment 28 Michel Normand 2018-08-16 17:52:05 UTC
my comment#27 is probably useless as not related to search key F7B333B196C8009B* of comment#26
As per key name I assume this is a key generated by mksusecd used to embed dud in iso, related mksusecd 
I do not know how mksusecd is supposed to embed this key in the generated iso.
May be Steffen ?

> + sudo mksusecd --verbose --verbose --create opensuse_20180816.iso --rebuild-initrd --initrd ./opensuse_20180816.dud --kernel ./k/kernel-default-4.17.14-20180816.gdc49b43.ppc64le.rpm ./k/kernel-firmware-20180730-1.1.noarch.rpm -- /home/michel/iso/openSUSE-Tumbleweed-DVD-ppc64le-Build679.1-Media.iso
> mksquashfs has '-comp': 1
> Repositories:
>   openSUSE [20180813-0]
> assuming repo-md sources
> added linuxrc option InstSys="disk:/boot/ppc64le/root"
> transient signing key created, keyid = F7B333B196C8009B
> signing key added to initrd
> kernel version: 4.17.14-1-default --> 4.17.14-20180816.gdc49b43-default
> re-signing '/CHECKSUMS'
> CHRP bootable (ppc64le)
> running:
> /usr/bin/mkisofs -U -l -r -pad -input-charset utf8 -o 'opensuse_20180816.iso' -V 'openSUSE-Tumbleweed-DVD-ppc64le6' -A 'openSUSE-Tumbleweed-DVD-ppc64le-Build679.1-Media' -p 'KIWI - http://opensuse.github.com/kiwi' -publisher 'SUSE LINUX GmbH' -J -f -joliet-long -exclude-list '/tmp/mksusecd.NrtdeYaT/exclude' -graft-points -path-list '/tmp/mksusecd.NrtdeYaT/filelist'
> building: 100%
> running:
> isohybrid  --no-code --no-chs --type 0x96 'opensuse_20180816.iso'
> isohybrid: Warning: more than 1024 cylinders: 3702
> isohybrid: Not all BIOSes will be able to boot this device
> calculating sha256...
Comment 29 Steffen Winterfeldt 2018-08-17 07:46:51 UTC
The key is in /usr/lib/rpm/gnupg/keys/ and in the /installkey.gpg keyring.
Comment 30 Steffen Winterfeldt 2018-08-17 08:36:14 UTC
Created attachment 780009 [details]
gpg key and signed CHECKSUMS file

I could reproduce this.

Here's the key and a CHECKSUMS file signed with it.
Comment 31 Michael Andres 2018-08-17 11:19:06 UTC
It seems to be caused by some change in libgpgme. My tests succeed with e.g. libgpgme11-1.9.0 and fail with libgpgme11-1.11.1 (which is also used here). Visible difference in the log (grep 'Determined key id'):

> Determined key id [B88B2FD43DBDC284] for signature /var/cache/zypp/raw/openSUSE-20180813-0lrUasm/repodata/repomd.xml.asc
> Determined key id [FA85E6362A1F940DD81D74A5F7B333B196C8009B] for signature /var/tmp/TmpDir.r4qmV1/CHECKSUMS.asc
> Determined key id [FA85E6362A1F940DD81D74A5F7B333B196C8009B] for signature /var/tmp/TmpDir.CIqLKI/CHECKSUMS.asc
> Determined key id [B88B2FD43DBDC284] for signature /mnt/var/cache/zypp/raw/repo-updateo2OHnr/repodata/repomd.xml.asc
> Determined key id [B88B2FD43DBDC284] for signature /mnt/var/cache/zypp/raw/repo-ossHEWlz4/repodata/repomd.xml.asc

For whatever reason for this key now the full fingerprint is returned and not the (shorter) id as for other keys. I'll check this with the libgpgme maintainers, Then we'll see how to fix it. Working around it in libvzypp is probably not a big deal, but we need to update....

I take the bug for 'libzypp'.
Comment 32 Michael Andres 2018-08-17 12:17:14 UTC
So it looks like it's not a matter of the key itself, but of the gpg version that was used to create the signature. libgpgme used to return the keyid 'F7B333B196C8009' of the key that created the signature.

Now with ibgpgme11-1.11.0 and if a recent gpg version was used to create the signature, the field may contain the full fingerprint, which brakes the key search.

> https://github.com/gpg/gpgme/commit/478d1650bbef84958ccce439fac982ef57b16cd0
> core: For a failed verification return the sig's fingerprint. 
> 
> This works only when the signatures features an ISSUER_FPR sub-packet
> and with GnuPG >= 2.2.7.  If that is not the case the keyid is kept in
> the FPR field.
Comment 33 Michael Andres 2018-08-17 12:58:16 UTC
Fixed in   
  TUMBLEWEED      libzypp 17. 6. 1   
  SLE-15          libzypp 17. 6. 1   

SLE-12* is not affected as it does not use libgpgme.
Comment 37 Swamp Workflow Management 2018-09-11 19:11:45 UTC
SUSE-SU-2018:2690-1: An update that solves two vulnerabilities and has 26 fixes is now available.

Category: security (important)
Bug References: 1036304,1041178,1043166,1045735,1058515,1066215,1070770,1070851,1082318,1084525,1088037,1088705,1091624,1092413,1093103,1096217,1096617,1096803,1099847,1100028,1100095,1100427,1101349,1102019,1102429,408814,428822,907538
CVE References: CVE-2017-9269,CVE-2018-7685
Sources used:
SUSE Linux Enterprise Module for Development Tools 15 (src):    libsolv-0.6.35-3.5.2
SUSE Linux Enterprise Module for Basesystem 15 (src):    libsolv-0.6.35-3.5.2, libzypp-17.6.4-3.10.1, zypper-1.14.10-3.7.1
Comment 38 Swamp Workflow Management 2018-09-17 10:11:42 UTC
openSUSE-SU-2018:2739-1: An update that solves two vulnerabilities and has 26 fixes is now available.

Category: security (important)
Bug References: 1036304,1041178,1043166,1045735,1058515,1066215,1070770,1070851,1082318,1084525,1088037,1088705,1091624,1092413,1093103,1096217,1096617,1096803,1099847,1100028,1100095,1100427,1101349,1102019,1102429,408814,428822,907538
CVE References: CVE-2017-9269,CVE-2018-7685
Sources used:
openSUSE Leap 15.0 (src):    libsolv-0.6.35-lp150.2.3.1, libzypp-17.6.4-lp150.2.3.1, zypper-1.14.10-lp150.2.3.1