Bug 1202116

Summary: kernel-default-devel = 5.3.18-57 missing in repo meta data of Leap 15.3 (was: nvidia update 470.129.06 => 470.141.03 fails)
Product: [openSUSE] openSUSE Distribution Reporter: Paul Tannington <paul.pgp-7>
Component: YaST2Assignee: Stefan Dirsch <sndirsch>
Status: RESOLVED FIXED QA Contact: Jiri Srain <jsrain>
Severity: Normal    
Priority: P3 - Medium CC: arvidjaar, joerg.kuehne, lubos.kocman, ma, mlin, paul.pgp-7, stvdo
Version: Leap 15.3   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
See Also: https://bugzilla.suse.com/show_bug.cgi?id=1205787
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 1205787    
Attachments: screenshot showing kernel-default-devel "versions"

Description Paul Tannington 2022-08-03 16:12:20 UTC
On openSUSE Leap 15.3 nvidia update (from https://download.nvidia.com/opensuse/leap/15.3) 470.129.06-lp153.57.1 => 470.141.03-lp153.53.1 fails due to:

"nothing provides 'kernel-default-devel = 5.3.18-57'"

The 15.3 kernel is currently at 5.3.18-150300.59.87.1


Full conflict log:

#### YaST2 conflicts list - generated 2022-08-03 16:58:10 ####

nothing provides 'kernel-default-devel = 5.3.18-57' needed by the to be installed nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64

    [ ] do not install nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64

    [ ] break nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64 by ignoring some of its dependencies


nothing provides 'kernel-default-devel = 5.3.18-57' needed by the to be installed nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64

    [ ] break nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64 by ignoring some of its dependencies

    [ ] do not install x11-video-nvidiaG05-470.141.03-lp153.53.1.x86_64


nothing provides 'kernel-default-devel = 5.3.18-57' needed by the to be installed nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64

    [ ] do not install nvidia-glG05-470.141.03-lp153.53.1.x86_64

    [ ] break nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64 by ignoring some of its dependencies


nothing provides 'kernel-default-devel = 5.3.18-57' needed by the to be installed nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64

    [ ] do not install nvidia-computeG05-470.141.03-lp153.53.1.x86_64

    [ ] break nvidia-gfxG05-kmp-default-470.141.03_k5.3.18_57-lp153.53.1.x86_64 by ignoring some of its dependencies



#### YaST2 conflicts list END ###
Comment 1 Stefan Dirsch 2022-08-04 02:18:00 UTC
This specific requirement has been introduced to fix boo#1201936.  kernel-default-devel = 5.3.18-57 is available via 

  http://download.opensuse.org/distribution/leap/15.3/repo/oss/x86_64/kernel-default-devel-5.3.18-57.3.x86_64.rpm

which needs kernel-devel = 5.3.18-57, which is available via

  http://download.opensuse.org/distribution/leap/15.3/repo/oss/noarch/kernel-devel-5.3.18-57.3.noarch.rpm

Could it be, that you no longer have the original installation repository enabled, which provide the initial versions of all needed packages? The output of 

  zypper repos -u

would be very useful in this context.
Comment 2 Paul Tannington 2022-08-04 08:01:49 UTC
Output from "zypper repos -u"



paul@Orion-17:~> zypper repos -u
Repository priorities are without effect. All enabled repositories share the same priority.

#  | Alias                                | Name                        | Enabled | GPG Check | Refresh | URI
---+--------------------------------------+-----------------------------+---------+-----------+---------+-------------------------------------------------------------------------
 1 | KDE-Extra                            | KDE Extra                   | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/repositories/KDE:/Extra/openSUSE_Leap_15.3
 2 | Local-SUSE-Repository                | Local Repository            | No      | ----      | ----    | dir:/home/paul/Local/Local-SUSE-Repository
 3 | Mozilla                              | Mozilla                     | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/repositories/mozilla/openSUSE_Leap_15.3/
 4 | backports                            | Update-Backports            | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/update/leap/15.3/backports/
 5 | download.opensuse.org-non-oss-main   | Main Repository (Non-OSS)   | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/distribution/leap/15.3/repo/non-oss/
 6 | download.opensuse.org-non-oss-update | Update Repository (Non-OSS) | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/update/leap/15.3/non-oss/
 7 | download.opensuse.org-oss-main       | Main Repository (OSS)       | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/distribution/leap/15.3/repo/oss/
 8 | download.opensuse.org-oss-update     | Update Repository (OSS)     | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/update/leap/15.3/oss
 9 | nVidia-Graphics-Driver               | nVidia Graphics Drivers     | Yes     | (r ) Yes  | Yes     | https://download.nvidia.com/opensuse/leap/15.3
10 | opensuse-guide.org-repo              | Libdvdcss Repository        | No      | ----      | ----    | http://opensuse-guide.org/repo/openSUSE_Leap_15.3/
11 | packman.inode.at-suse                | Packman                     | Yes     | (r ) Yes  | Yes     | http://ftp.gwdg.de/pub/linux/misc/packman/suse/openSUSE_Leap_15.3/
12 | sle                                  | Update-SLE                  | Yes     | (r ) Yes  | Yes     | https://download.opensuse.org/update/leap/15.3/sle/
paul@Orion-17:~>
Comment 3 Paul Tannington 2022-08-04 08:14:35 UTC
Created attachment 860596 [details]
screenshot showing kernel-default-devel "versions"

http://download.opensuse.org/distribution/leap/15.3/repo/oss/ is enabled, as repo #7

Screenshot of Yast2 software_single showing "versions" tab for kernel-default-devel
Comment 4 Stefan Dirsch 2022-08-04 08:25:44 UTC
Indeed. http://download.opensuse.org/distribution/leap/15.3/repo/oss/  repo is enabled and I have no idea, why this version is not found by zypper/YaST.

Of course you can download both packages manually and install them via

zypper in \
kernel-default-devel-5.3.18-57.3.x86_64.rpm
kernel-devel-5.3.18-57.3.noarch.rpm

before doing the nvidia driver update.
Comment 5 Paul Tannington 2022-08-04 08:52:28 UTC
Prior to manually installing I performed a zypper refresh (again), which made no difference to zypper/yast finding the packages.

Manually installed both 5.3.18-57.3 packages, a subsequent zypper up installed the nvidia packages with no problem.

Thanks for your swift response to this issue.
Comment 6 Stefan Dirsch 2022-08-04 09:08:14 UTC
Thanks for confirmation (that this workaround works for you)!
Comment 7 Stefan Dirsch 2022-08-04 11:38:13 UTC
I can reproduce this issue on a fresh Leap 15.3 installation. I have no idea why YaST/zypper doesn't list 

  kernel-default-devel 5.3.18-57.3

It's in http://download.opensuse.org/distribution/leap/15.3/repo/oss/x86_64/

  kernel-devel

is in http://download.opensuse.org/distribution/leap/15.3/repo/oss/noarch/

# zypper repos -u|grep repo-oss
10 | repo-oss                        | Main Repository      | Yes     | (r ) Yes  | Yes     | http://download.opensuse.org/distribution/leap/15.3/repo/oss/

#  # zypper packages repo-oss|grep kernel-devel
v  | Main Repository | kernel-devel                                               | 5.3.18-59.10.1                                                    | noarch
   | Main Repository | kernel-devel-azure                                         | 5.3.18-38.8.1                                                     | noarch
   | Main Repository | kernel-devel-rt                                            | 5.3.18-8.13.1                                                     | noarch
# zypper packages repo-oss|grep kernel-default-devel
v  | Main Repository | kernel-default-devel                                       | 5.3.18-59.10.1                                                    | x86_64
Comment 8 Stefan Dirsch 2022-08-04 11:41:24 UTC
And yes, this 5.3.18-57.3 package *does* also provide 5.3.18-57 (without .3 suffix).

# rpm --provides -qp kernel-default-devel-5.3.18-57.3.x86_64.rpm | grep kernel-default-devel
kernel-default-devel = 5.3.18-57
kernel-default-devel = 5.3.18-57.3
kernel-default-devel(x86-64) = 5.3.18-57.3

# rpm --provides -qp kernel-devel-5.3.18-57.3.noarch.rpm
kernel-devel = 5.3.18-57
kernel-devel = 5.3.18-57.3
Comment 9 Stefan Dirsch 2022-08-04 12:00:17 UTC
Reassigning to YaST component. Adding also our zypper/libzypp expert.
Comment 12 Paul Tannington 2022-08-04 12:47:11 UTC
(In reply to Stefan Dirsch from comment #7)
> I can reproduce this issue on a fresh Leap 15.3 installation. I have no idea
> why YaST/zypper doesn't list 
> 
>   kernel-default-devel 5.3.18-57.3
> 

Out of curiosity. I just looked at a second machine I have which is also still on Leap 15.3, (which has amd graphics so I'd not seen the issue), YaST/zypper doesn't list the 5.3.18-57.3 packages on this machine either.
Comment 13 Andrei Borzenkov 2022-08-04 13:30:39 UTC
(In reply to Stefan Dirsch from comment #9)
> Reassigning to YaST component. 

It has nothing to do with YaST. This kernel does not exist in repository metadata.

bor@leap15:~> dumpsolv /var/cache/zypp/solv/download.opensuse.org-oss/solv | grep -E 'solvable:(name|arch|evr):' | grep -A 2 kernel-default-devel | grep x86_64 -A1 -B1
solvable:name: kernel-default-devel
solvable:arch: x86_64
solvable:evr: 5.3.18-59.10.1
bor@leap15:~> zcat /var/cache/zypp/raw/download.opensuse.org-oss/repodata/fd158f7e9bd24549ba5cf7c2b34157dc1b9298df86bd0505477f4ac5f2f155f4-primary.xml.gz | grep -F 5.3.18-57
bor@leap15:~>
Comment 14 Stefan Dirsch 2022-08-04 13:56:54 UTC
 Thanks for figuring out, Andrei! So may I ask why this is missing from repo metadata?  To whom can I assign this ticket?
Comment 15 Andrei Borzenkov 2022-08-04 14:02:38 UTC
(In reply to Stefan Dirsch from comment #14)
> So may I ask why this is missing from repo metadata?

How Leap is assembled since 15.3 is complete mystery to me.

> To whom can I assign this ticket?

Lubos Kocman as Leap release manager?
Comment 16 Stefan Dirsch 2022-08-04 14:19:42 UTC
Thanks a lot, Andrei! I've contacted him now ...
Comment 17 Lubos Kocman 2022-08-04 14:29:53 UTC
Am I reading correctly that we need to republish repodata for 15.3 GA oss repo as on of rpms physically available there is not part of repodata.

Also all updates of kernel should come from the sle update repository, as both kernel and kernel-devel is formally inherited as a binary from SLES 15 SP3 (Update). I expect there was at least one kernel udpate since GA.

That repository is https://download.opensuse.org/update/leap/15.3/sle/

Just to eliminate some of my thoughts... Do you have

vendors=openSUSE,SUSE,SUSE LLC <https://www.suse.com/>

in /etc/zypp/vendors.d/00-openSUSE.conf ?

As in 15.3 we basically made SUSE/openSUSE vendors interchangable without --allow-vendor-change option.

Thank you
Comment 18 Stefan Dirsch 2022-08-04 14:45:46 UTC
On my affected system I have

[main]
vendors=openSUSE,SUSE,SUSE LLC <https://www.suse.com/>

in /etc/zypp/vendors.d/00-openSUSE.conf.
Comment 19 Stefan Dirsch 2022-08-04 14:56:07 UTC
(In reply to Lubos Kocman from comment #17)
> Am I reading correctly that we need to republish repodata for 15.3 GA oss
> repo as one of rpms physically available there is not part of repodata.

Yes, I would say so. Actually two.

  kernel-default-devel-5.3.18-57.3.x86_64.rpm
  kernel-devel-5.3.18-57.3.noarch.rpm

are missing in metadata for

   http://download.opensuse.org/distribution/leap/15.3/repo/oss

repository.

> Also all updates of kernel should come from the sle update repository, as
> both kernel and kernel-devel is formally inherited as a binary from SLES 15
> SP3 (Update). I expect there was at least one kernel udpate since GA.
> 
> That repository is https://download.opensuse.org/update/leap/15.3/sle/

Yes, but all these update kernels already have -59 versions. I don't think metadata is missing there. I've seen a lot of these update kernels in "zypper packages" output for this sle repo.
Comment 20 Lubos Kocman 2022-08-04 15:01:30 UTC
I'll talk to Max tomorrow about safest way to do so, I believe we were already republishing the trees as part of the media Quaterly Update respin, which could be also the root cause.
Comment 21 Stefan Dirsch 2022-08-04 15:08:26 UTC
JFYI, this issue does not occur on Leap 15.4, because there is only one kernel version (5.14.21-150400.22.1)  below

http://download.opensuse.org/distribution/leap/15.4/repo/oss/
Comment 22 Stefan Dirsch 2022-08-04 15:13:16 UTC
(In reply to Lubos Kocman from comment #20)
> I'll talk to Max tomorrow about safest way to do so, I believe we were
> already republishing the trees as part of the media Quaterly Update respin,
> which could be also the root cause.

Thanks! I don't think the Quaterly Update respin was the issue here. I guess this was missing before in repo metadate.W e just uncovered it with the new requirement for the GA kernel now. This is new since yesterday.
Comment 23 Paul Tannington 2022-08-04 16:02:29 UTC
(In reply to Lubos Kocman from comment #17)

> 
> Just to eliminate some of my thoughts... Do you have
> 
> vendors=openSUSE,SUSE,SUSE LLC <https://www.suse.com/>
> 
> in /etc/zypp/vendors.d/00-openSUSE.conf ?
> 

Just to confirm, yes, the same as @stefan posted in #18

paul@Orion-17:~> cat /etc/zypp/vendors.d/00-openSUSE.conf
[main]
vendors=openSUSE,SUSE,SUSE LLC <https://www.suse.com/>
Comment 24 Stefan Dirsch 2022-08-05 13:58:10 UTC
Leap 15.4/sle15-sp4: not affected by that issue; only one version in GA repo
sle15-sp3: not affected by that issue; only one version  (5.3.18-57.3)  in GA repo
Comment 26 Stefan Dirsch 2022-08-05 14:17:22 UTC
sle15-sp2: not affected by that issue; only one version (5.3.18-22.2) in GA repo
Comment 27 Stefan Dirsch 2022-08-05 14:56:13 UTC
sle12-sp5: not affected by that issue; only one version (4.12.14-120.1) in GA repo
Comment 28 Paul Tannington 2022-08-06 11:07:58 UTC
From a post ( https://forums.opensuse.org/showthread.php/?p=3147815#post3147815 ) on the openSUSE forums, this issue is also seen with Tumbleweed.
Comment 29 Andrei Borzenkov 2022-08-06 11:45:38 UTC
(In reply to Paul Tannington from comment #28)
> From a post (
> https://forums.opensuse.org/showthread.php/?p=3147815#post3147815 ) on the
> openSUSE forums, this issue is also seen with Tumbleweed.

It is unrelated to this bug report. The http://bugzilla.opensuse.org/show_bug.cgi?id=1201936 does not apply to Tumbleweed at all (which does not have "GA kernel"), but this change made nvidia KMP require exact kernel version present at package build time. 

Please open new bug report for Tumbleweed, reference http://bugzilla.opensuse.org/show_bug.cgi?id=1201936.
Comment 30 Paul Tannington 2022-08-06 12:21:02 UTC
OK, I clearly misunderstood, thanks for the clarification and I apologise for the noise.
Comment 31 Stefan Dirsch 2022-08-06 18:33:52 UTC
(In reply to Andrei Borzenkov from comment #29)
> (In reply to Paul Tannington from comment #28)
> > From a post (
> > https://forums.opensuse.org/showthread.php/?p=3147815#post3147815 ) on the
> > openSUSE forums, this issue is also seen with Tumbleweed.
> 
> It is unrelated to this bug report. The
> http://bugzilla.opensuse.org/show_bug.cgi?id=1201936 does not apply to
> Tumbleweed at all (which does not have "GA kernel"), but this change made
> nvidia KMP require exact kernel version present at package build time. 

Indeed. I did sth. stupid :-(. This change should have not been done for TW. Fixed this now, but packages still need to be updated on 
nvidia's server.

-------------------------------------------------------------------
Sat Aug  6 18:15:17 UTC 2022 - Stefan Dirsch <sndirsch@suse.com>

- no longer try to require a "GA" kernel-devel package on TW; 
  there is no such thing on TW; it has been an oversight to do
  the previous change below also for TW (boo#1202192)

> Please open new bug report for Tumbleweed, reference
> http://bugzilla.opensuse.org/show_bug.cgi?id=1201936.

https://bugzilla.opensuse.org/show_bug.cgi?id=1202192
Comment 32 Max Lin 2022-08-08 07:49:13 UTC
(In reply to Andrei Borzenkov from comment #15)
> (In reply to Stefan Dirsch from comment #14)
> > So may I ask why this is missing from repo metadata?
> 
> How Leap is assembled since 15.3 is complete mystery to me.
> 
> > To whom can I assign this ticket?
> 
> Lubos Kocman as Leap release manager?

Is nvidia drivers builds against GA project? If so it should switch to use openSUSE:Leap:15.3:Update:Respin as the build-base, starting from Leap 15.3 that Leap does enabled quarterly update, that means QU update would've including packages from :Update, like kernel in this case, repomd got updated follows with QU update, that's why 5.3.18-59 has listed in repomd, not 5.3.18-57(GA) anymore.
Comment 33 Lubos Kocman 2022-08-08 09:31:38 UTC
I was told by Stefan that he would be back at the issue that he wanted to avoid https://bugzilla.opensuse.org/show_bug.cgi?id=1201936 

which is explicitly why he decided to build against :GA
Comment 34 Stefan Dirsch 2022-08-20 12:53:23 UTC
(In reply to Lubos Kocman from comment #33)
> I was told by Stefan that he would be back at the issue that he wanted to
> avoid https://bugzilla.opensuse.org/show_bug.cgi?id=1201936 
> 
> which is explicitly why he decided to build against :GA

Indeed, I would be back to this issue by building against :Update kernel-devel (and requiring it). I don't understand why we exclude repo content from the meta files. What would be the harm adding it again to the meta files?
Comment 35 Lubos Kocman 2022-08-22 11:22:30 UTC
I did email autobuild team regarding option to republish content of 15.3 GA from obs to ftp-stage. As the current ftp-stage content also contains QU2 content (isos+rpms). I'll keep the bug posted.

In order to prevent this in the future we need to not republish the GA rpm-repos for 15.4. Also, removing the leap15.4_qu1 tag as this is 15.3 issue.

We'll have to revisit:
https://en.opensuse.org/openSUSE:LeapQuarterlyUpdates_howto
Comment 36 Stefan Dirsch 2022-08-22 11:45:49 UTC
Thanks for your efforts, Lubos! So setting also NEEDINFO to you. ;-)

Luckily, we haven't seen that issue (yet) for 

Leap 15.4
sle12-sp5
sle15-sp2
sle15-sp3
sle15-sp4

I have checked this. See my comments #24,26,27
Comment 37 Steve Dobrinski 2022-08-24 11:33:56 UTC
I have difficulties to understand how changes to the GA repositories should get around this dependency problem: Even if 'kernel-devel' and 'kernel-default-devel' packages were published in GA in different versions, a simple system update would always change them to the most up-to-date version which would then be incompatible with Nvidia driver builds. (Re-)Installation of Nvidia RPMs would then require a (forced) downgrade of the installed kernel-devel and kernel-default-devel packages, because there can be only one version of these packages in the system. The next system update would then update kernel-devel and kernel-default-devel again ...

If kernel ABIs nowadays may break with newer versions of the kernel, there is no way to use modules compiled at random system states for all kernel versions. 

Maybe switch to DKMS consequently and build the modules for every single kernel version instead of trying to link modules around?
Comment 38 Stefan Dirsch 2022-08-24 12:09:20 UTC
(In reply to Steve Dobrinski from comment #37)
> I have difficulties to understand how changes to the GA repositories should
> get around this dependency problem: Even if 'kernel-devel' and
> 'kernel-default-devel' packages were published in GA in different versions,
> a simple system update would always change them to the most up-to-date
> version which would then be incompatible with Nvidia driver builds.

And? The nvidia kernel modules are still built with the right GA version. Nothing will change
by updating kernel-devel and kernel-default-devel.

> (Re-)Installation of Nvidia RPMs would then require a (forced) downgrade of
> the installed kernel-devel and kernel-default-devel packages, because there
> can be only one version of these packages in the system. 

Which is wrong. You can have several version installed at the same time.

# rpm -qa|grep kernel-devel
kernel-devel-5.14.21-150400.24.11.1.noarch
# rpm -qa|grep kernel-default-devel
kernel-default-devel-5.14.21-150400.24.11.1.x86_64

# zypper in kernel-default-devel
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 2 NEW packages are going to be installed:
  kernel-default-devel-5.14.21-150400.24.18.1
  kernel-devel-5.14.21-150400.24.18.1

2 new packages to install.
Overall download size: 30,7 MiB. Already cached: 0 B. After the operation,
additional 62,7 MiB will be used.
[]
# rpm -qa|grep kernel-devel
kernel-devel-5.14.21-150400.24.18.1.noarch
kernel-devel-5.14.21-150400.24.11.1.noarch
# rpm -qa|grep kernel-default-devel
kernel-default-devel-5.14.21-150400.24.11.1.x86_64
kernel-default-devel-5.14.21-150400.24.18.1.x86_64

> The next system
> update would then update kernel-devel and kernel-default-devel again ...

I'm not sure but it would not matter.

> If kernel ABIs nowadays may break with newer versions of the kernel, there
> is no way to use modules compiled at random system states for all kernel
> versions. 

Depends on how you define kABI breakage. If symbols are just added, what was the issue her,
things can be fixed by always building against the GA version.

> Maybe switch to DKMS consequently and build the modules for every single
> kernel version instead of trying to link modules around?

Feel free to discuss this in a separate ticket or on your favorite openSUSE mailing list.
Comment 39 Steve Dobrinski 2022-08-24 14:13:40 UTC
(In reply to Stefan Dirsch from comment #38)
> (In reply to Steve Dobrinski from comment #37)
> > (Re-)Installation of Nvidia RPMs would then require a (forced) downgrade of
> > the installed kernel-devel and kernel-default-devel packages, because there
> > can be only one version of these packages in the system. 
> 
> Which is wrong. You can have several version installed at the same time.

I stand corrected. I was mislead by zypper messages when not using '--oldpackage'. By these I assumed that zypper would downgrade the installed packages. With '--oldpackage' older versions of the kernel development packages are installed in parallel.
Comment 40 Jörg Kühne 2022-09-01 08:20:09 UTC
What is the proposed solution for this issue now?

1) Wait for an official fix from SUSE for the metadata issue
2) Install the needed packages manually (see Comment 4)

Thank you,

Jörg
Comment 41 Stefan Dirsch 2022-09-01 09:55:46 UTC
(In reply to Jörg Kühne from comment #40)
> What is the proposed solution for this issue now?
> 
> 1) Wait for an official fix from SUSE for the metadata issue
> 2) Install the needed packages manually (see Comment 4)

For now you need to use the workaround from comment#4. Metadata issue is still work-in-progress/in discussion. :-(
Comment 42 Lubos Kocman 2022-09-07 07:44:32 UTC
Fixed by republishing 15.3 repos on ftp-prod and ftp-stage.

Sorry for the delay.

mirror@pontifex2:~/openSUSE-release-tools> ls -lad /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/*/kernel*devel*.rpm
-rw-r--r-- 2 mirror stage  9157588 May  7  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-64kb-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage  7737196 May  7  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-64kb-livepatch-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage  9170812 May  7  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-default-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage  9155996 May  7  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-preempt-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage 19663919 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/noarch/kernel-devel-5.3.18-57.3.noarch.rpm
-rw-r--r-- 2 mirror stage 19688505 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/noarch/kernel-devel-azure-5.3.18-36.1.noarch.rpm
-rw-r--r-- 2 mirror stage 16016215 Feb  4  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/noarch/kernel-devel-rt-5.3.18-8.3.1.noarch.rpm
-rw-r--r-- 2 mirror stage  8009416 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/ppc64le/kernel-debug-devel-5.3.18-57.3.ppc64le.rpm
-rw-r--r-- 2 mirror stage  8006204 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/ppc64le/kernel-default-devel-5.3.18-57.3.ppc64le.rpm
-rw-r--r-- 2 mirror stage  7132760 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/ppc64le/kernel-kvmsmall-devel-5.3.18-57.3.ppc64le.rpm
-rw-r--r-- 2 mirror stage  7140424 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/s390x/kernel-default-devel-5.3.18-57.3.s390x.rpm
-rw-r--r-- 2 mirror stage  7319712 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-azure-devel-5.3.18-36.1.x86_64.rpm
-rw-r--r-- 2 mirror stage  8980612 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-debug-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  8978292 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-default-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  7271592 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-kvmsmall-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  8976980 May  6  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-preempt-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  5178812 Feb  4  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-rt_debug-devel-5.3.18-8.3.1.x86_64.rpm
-rw-r--r-- 2 mirror stage  5146332 Feb  4  2021 /srv/ftp-stage/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-rt-devel-5.3.18-8.3.1.x86_64.rpm
mirror@pontifex2:~/openSUSE-release-tools> ls -lad /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/*/kernel*devel*.rpm
-rw-r--r-- 2 mirror stage  9157588 May  7  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-64kb-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage  7737196 May  7  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-64kb-livepatch-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage  9170812 May  7  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-default-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage  9155996 May  7  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/aarch64/kernel-preempt-devel-5.3.18-57.3.aarch64.rpm
-rw-r--r-- 2 mirror stage 19663919 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/noarch/kernel-devel-5.3.18-57.3.noarch.rpm
-rw-r--r-- 2 mirror stage 19688505 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/noarch/kernel-devel-azure-5.3.18-36.1.noarch.rpm
-rw-r--r-- 2 mirror stage 16016215 Feb  4  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/noarch/kernel-devel-rt-5.3.18-8.3.1.noarch.rpm
-rw-r--r-- 2 mirror stage  8009416 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/ppc64le/kernel-debug-devel-5.3.18-57.3.ppc64le.rpm
-rw-r--r-- 2 mirror stage  8006204 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/ppc64le/kernel-default-devel-5.3.18-57.3.ppc64le.rpm
-rw-r--r-- 2 mirror stage  7132760 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/ppc64le/kernel-kvmsmall-devel-5.3.18-57.3.ppc64le.rpm
-rw-r--r-- 2 mirror stage  7140424 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/s390x/kernel-default-devel-5.3.18-57.3.s390x.rpm
-rw-r--r-- 2 mirror stage  7319712 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-azure-devel-5.3.18-36.1.x86_64.rpm
-rw-r--r-- 2 mirror stage  8980612 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-debug-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  8978292 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-default-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  7271592 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-kvmsmall-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  8976980 May  6  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-preempt-devel-5.3.18-57.3.x86_64.rpm
-rw-r--r-- 2 mirror stage  5178812 Feb  4  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-rt_debug-devel-5.3.18-8.3.1.x86_64.rpm
-rw-r--r-- 2 mirror stage  5146332 Feb  4  2021 /srv/ftp/pub/opensuse//distribution/leap/15.3/repo/oss/x86_64/kernel-rt-devel-5.3.18-8.3.1.x86_64.rpm
mirror@pontifex2:~/openSUSE-release-tools>
Comment 43 Stefan Dirsch 2022-09-07 12:21:10 UTC
Thanks a lot! I just verified that things are working now, i.e. kernel-default-devel and kernel-devel of GA kernel are autoselected and kernel modules are built against it - and weak modules symlinks created for the update kernel.

I noticed that apparently for Leap 15.3 we no longer offer the nVidia repo as Community repository. I'll open a separate ticket for this.
Comment 44 Stefan Dirsch 2022-09-07 13:31:24 UTC
(In reply to Stefan Dirsch from comment #43)
> I noticed that apparently for Leap 15.3 we no longer offer the nVidia repo
> as Community repository. I'll open a separate ticket for this.

--> boo#1203213