Bug 1198534

Summary:	Raspi 4: boots to blank screen with kernel 5.3.18-150300.59.63-default
Product:	[openSUSE] openSUSE Distribution	Reporter:	Rolf Wirth <ro.wirth>
Component:	Kernel	Assignee:	Ivan Ivanov <ivan.ivanov>
Status:	RESOLVED FIXED	QA Contact:	E-mail List <qa-bugs>
Severity:	Critical
Priority:	P3 - Medium	CC:	angel, axel.braun, axel.keller, dheidler, fvogt, ivan.ivanov, llrainey15, martin.weiss, mbrugger, meissner, patrik.jakobsson, tiwai
Version:	Leap 15.3
Target Milestone:	---
Hardware:	aarch64
OS:	Other
Whiteboard:
Found By:	---	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---
Attachments:	Output of dmesg with working kernel 59.60 Output of dmesg with failing kernel 59.63 Kernel-default-5.3.18-150300.1.1.g27e7898.aarch64.rpm installation and testing

Description Rolf Wirth 2022-04-16 11:29:36 UTC

Updated Kernel from 5.3.18-150300.59.60-default to 5.3.18-150300.59.63-default.
Now system boots to a blank screen after showing some boot messages (until kernel load modules?).
System is running (can ssh to it) but console it unusable.
Selecting old kernel during grub2 all is OK.
How can I solve this?
Difference of loaded modules:
 # diff mod_60 mod_63
34c34
< drm_kms_helper        258048  3 vc4
---
> drm_kms_helper        258048  1 vc4
101c101
< vc4                   249856  6
---
> vc4                   249856  7

Comment 1 Ivan Ivanov 2022-04-21 06:57:43 UTC

What monitor do you have attached to the device, resolution?

Comment 2 Ivan Ivanov 2022-04-21 07:04:20 UTC

Please, could you save and attach here dmesg output. Thanks!

Comment 3 Rolf Wirth 2022-04-21 09:18:34 UTC

Created attachment 858302 [details]
Output of dmesg with working kernel 59.60

Comment 4 Rolf Wirth 2022-04-21 09:21:28 UTC

I'm using a LG 24MB56HQ-B as monitor with a resolution of 1920x1080
Remember, monitor is working with kernel 59.60 but fails with 59.63
I attached dmesg_59.60.txt  dmesg_59.63.txt

Comment 5 Rolf Wirth 2022-04-21 09:23:25 UTC

Created attachment 858303 [details]
Output of dmesg with failing kernel 59.63

Comment 6 Patrik Jakobsson 2022-04-27 08:57:00 UTC

This is a possible regression from backporting:

bca10db67bda drm/vc4: crtc: Make sure the HDMI controller is powered when disabling

suse-commit: 0e082ec0cb77c1d3c1bbfba5c15215c5cb5f9fd4

I'm building a test kernel at:
https://build.opensuse.org/project/show/home:patrikjakobsson:bsc1198534

Comment 7 Rolf Wirth 2022-04-27 19:52:41 UTC

@ Patrik
Can "I" help in any way?
Testing? Where can I get this test kernel as rpm?

Comment 8 Ivan Ivanov 2022-04-29 12:20:37 UTC

*** Bug 1198524 has been marked as a duplicate of this bug. ***

Comment 9 Angel Lafuente 2022-04-30 13:12:11 UTC

Hello everyone:

I can confirm the issue on a Raspberry 400.

I found two easy workarounds meanwhile the bug is fixed.

1- Hardware way:

- Unplug the HDMI wire from Raspberry.
- Power on the Raspberry. Wait two minutos or so to Linux start up.
- Plug the HDMI wire in the Raspberry.
- Power on your monitor.
- If Linux is on rescue mode, login as 'root' and execute 'init 5' to reach to graphical mode.

2- Software way:

On boot GRUB options > advanced options > choose kernel-default-5.3.18-150300.59.60.4.

Prevent this kernel to be remove on future updates: systemctl disable purge-kernels.

Hope this helps.

Best regards,

Comment 10 Ivan Ivanov 2022-05-03 06:30:50 UTC

Hi, I believe I found the issue. Please test the following [1]
kernel package it fixes the issue for me.

[1] https://download.opensuse.org/repositories/home:/iivanov:/fix-vc4-crtc-bsc1198534/pool/aarch64/kernel-default-5.3.18-150300.1.1.g27e7898.aarch64.rpm

Comment 11 Ivan Ivanov 2022-05-03 07:07:49 UTC

*** Bug 1198527 has been marked as a duplicate of this bug. ***

Comment 12 Ivan Ivanov 2022-05-03 07:14:39 UTC

*** Bug 1199066 has been marked as a duplicate of this bug. ***

Comment 13 Larry Rainey 2022-05-03 13:13:53 UTC

It works on my Pi4

Comment 14 Angel Lafuente 2022-05-03 18:25:32 UTC

Created attachment 858620 [details]
Kernel-default-5.3.18-150300.1.1.g27e7898.aarch64.rpm installation and testing

Hello:

I installed the new kernel RPM, but I was forced to use the 'oldpackage' option and to ignore the unsigned RPM warning.

sudo zypper install --oldpackage ./kernel-default-5.3.18-150300.1.1.g27e7898.aarch64.rpm

If I choose the new kernel on boot,it works as it worked with the previous default kernel, 5.3.18-150300.59.60.4

I upload a brief attachment with further details.

Thank you

Comment 15 Ivan Ivanov 2022-05-03 19:13:07 UTC

(In reply to Angel Lafuente from comment #14)

> I installed the new kernel RPM, but I was forced to use the 'oldpackage'
> option and to ignore the unsigned RPM warning.

Ah, yes, I forget to mention it. This is not official kernel release,
this is just test kernel created in my OBS home project just to verify
that fix is working. Meanwhile I have sent "pull request" with the same
fix to the kernel maintainers. Hopefully it will be included in the
next kernel version.

Comment 16 Ivan Ivanov 2022-05-04 13:35:42 UTC

Fix was merged into SLE15-SP3 branch. Marking as fixed.

Comment 18 Axel Braun 2022-05-05 12:17:07 UTC

(In reply to Ivan Ivanov from comment #10)
> Hi, I believe I found the issue. Please test the following [1]
> kernel package it fixes the issue for me.
> 
> [1]
> https://download.opensuse.org/repositories/home:/iivanov:/fix-vc4-crtc-
> bsc1198534/pool/aarch64/kernel-default-5.3.18-150300.1.1.g27e7898.aarch64.rpm

I did an upgrade to Leap 15.3, and installed your kernel-package on top. Same result, black screen as before

Comment 19 Ivan Ivanov 2022-05-05 13:43:55 UTC

(In reply to Axel Braun from comment #18)
> (In reply to Ivan Ivanov from comment #10)
> > Hi, I believe I found the issue. Please test the following [1]
> > kernel package it fixes the issue for me.
> > 
> > [1]
> > https://download.opensuse.org/repositories/home:/iivanov:/fix-vc4-crtc-
> > bsc1198534/pool/aarch64/kernel-default-5.3.18-150300.1.1.g27e7898.aarch64.rpm
> 
> I did an upgrade to Leap 15.3, and installed your kernel-package on top.
> Same result, black screen as before

Just to be sure. Have you selected this kernel from the Grub menu list?

Comment 20 Axel Braun 2022-05-05 14:37:46 UTC

(In reply to Ivan Ivanov from comment #19)

> > I did an upgrade to Leap 15.3, and installed your kernel-package on top.
> > Same result, black screen as before
> 
> Just to be sure. Have you selected this kernel from the Grub menu list?

I cant confirm what the Raspi bootet (and I will not be able to check before next week)
Do you have an indication when the update will be available through the update channel?

Comment 21 Ivan Ivanov 2022-05-05 15:03:12 UTC

(In reply to Axel Braun from comment #20)
> (In reply to Ivan Ivanov from comment #19)
> 
> > > I did an upgrade to Leap 15.3, and installed your kernel-package on top.
> > > Same result, black screen as before
> > 
> > Just to be sure. Have you selected this kernel from the Grub menu list?
> 
> I cant confirm what the Raspi bootet (and I will not be able to check before
> next week)

My questions was whether you have explicitly selected 5.3.18-150300.1.1.g27e7898 kernel from Grub, because if not it will boot the other one, so you will
still observe the vc4 issue.

> Do you have an indication when the update will be available through the
> update channel?

Yep, just don't know when, but it should be soonish.

Comment 23 Rolf Wirth 2022-05-06 10:59:14 UTC

It works for me.
I installed the new kernel as written in Comment #14.
At boot/grub I selected this new kernel and it works.
Thanks

Comment 26 Axel Braun 2022-05-13 07:33:11 UTC

reply to Ivan Ivanov from comment #21)

> > Do you have an indication when the update will be available through the
> > update channel?
> 
> Yep, just don't know when, but it should be soonish.

Reopening as this still is not fixed, and people are running into issues here. Who triggers the update? Is this you, Ivan?

Comment 27 Ivan Ivanov 2022-05-13 07:54:17 UTC

(In reply to Axel Braun from comment #26)

> 
> Reopening as this still is not fixed, and people are running into issues
> here. Who triggers the update? Is this you, Ivan?

No, I am not triggering updates. Have you tried rpm from comment#10?
Please, follow same steps as Angel in comment#14 and make sure you have
selected this new kernel in grub menu.

Comment 28 Axel Braun 2022-05-13 08:17:53 UTC

(In reply to Ivan Ivanov from comment #27)
> (In reply to Axel Braun from comment #26)
> 
> > 
> > Reopening as this still is not fixed, and people are running into issues
> > here. Who triggers the update? Is this you, Ivan?
> 
> No, I am not triggering updates. Have you tried rpm from comment#10?
> Please, follow same steps as Angel in comment#14 and make sure you have
> selected this new kernel in grub menu.

Yes, I got it working for myself, but it needs to be in the mainline kernel ASAP.
cc'ing Markus

Comment 29 Axel Braun 2022-05-13 12:48:23 UTC

I have increased Prio and severity, as this leaves users with an unusable machine. Fix needs to be deployed urgently. Thank you

Comment 30 Fabian Vogt 2022-05-13 13:18:54 UTC

FWICT the fix is part of SUSE:Maintenance:24033, which has a planned release date of 2022-05-19 (next Thursday).

Comment 31 Takashi Iwai 2022-05-13 13:53:43 UTC

Please don't touch the priority field of Bugzilla.  It's only for developer side.

Comment 32 Axel Braun 2022-05-13 14:20:56 UTC

(In reply to Fabian Vogt from comment #30)
> FWICT the fix is part of SUSE:Maintenance:24033, which has a planned release
> date of 2022-05-19 (next Thursday).

Sorry to be a pest, but if we claim that oS should Enterprise SLES quality, we cant have a non-working kernel on one architecture for more than 4 weeks

Or we must redefine 'Enterprise Quality' to the level of.... (I was supposed to say M$-crap, But *I did not say it*!)

Comment 33 Marcus Meissner 2022-05-16 09:57:13 UTC

the 15-sp3 / 15.3 kernel release is 6 days overdue, we are poking QA already.

Regular schedule is kernel releases on SECOND tuesday of every month.

Comment 34 Swamp Workflow Management 2022-05-16 13:24:39 UTC

SUSE-SU-2022:1669-1: An update that solves 16 vulnerabilities, contains 6 features and has 29 fixes is now available.

Category: security (important)
Bug References: 1028340,1071995,1137728,1152472,1152489,1177028,1179878,1182073,1183723,1187055,1191647,1193556,1193842,1194625,1195651,1195926,1196018,1196114,1196367,1196514,1196639,1196942,1197157,1197391,1197656,1197660,1197677,1197914,1197926,1198077,1198217,1198330,1198400,1198413,1198437,1198448,1198484,1198515,1198516,1198534,1198742,1198825,1198989,1199012,1199024
CVE References: CVE-2020-27835,CVE-2021-0707,CVE-2021-20292,CVE-2021-20321,CVE-2021-38208,CVE-2021-4154,CVE-2022-0812,CVE-2022-1158,CVE-2022-1280,CVE-2022-1353,CVE-2022-1419,CVE-2022-1516,CVE-2022-28356,CVE-2022-28748,CVE-2022-28893,CVE-2022-29156
JIRA References: SLE-13208,SLE-13513,SLE-15172,SLE-15175,SLE-18234,SLE-8449
Sources used:
SUSE Linux Enterprise Realtime Extension 15-SP3 (src):    release-notes-sle_rt-15.3.20220422-150300.3.3.2
SUSE Linux Enterprise Module for Realtime 15-SP3 (src):    kernel-rt-5.3.18-150300.88.2, kernel-rt_debug-5.3.18-150300.88.2, kernel-source-rt-5.3.18-150300.88.2, kernel-syms-rt-5.3.18-150300.88.1, release-notes-sle_rt-15.3.20220422-150300.3.3.2
SUSE Linux Enterprise Micro 5.2 (src):    kernel-rt-5.3.18-150300.88.2
SUSE Linux Enterprise Micro 5.1 (src):    kernel-rt-5.3.18-150300.88.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

Comment 35 Swamp Workflow Management 2022-05-16 16:24:56 UTC

SUSE-SU-2022:1687-1: An update that solves 16 vulnerabilities, contains 6 features and has 29 fixes is now available.

Category: security (important)
Bug References: 1028340,1071995,1137728,1152472,1152489,1177028,1179878,1182073,1183723,1187055,1191647,1193556,1193842,1194625,1195651,1195926,1196018,1196114,1196367,1196514,1196639,1196942,1197157,1197391,1197656,1197660,1197677,1197914,1197926,1198077,1198217,1198330,1198400,1198413,1198437,1198448,1198484,1198515,1198516,1198534,1198742,1198825,1198989,1199012,1199024
CVE References: CVE-2020-27835,CVE-2021-0707,CVE-2021-20292,CVE-2021-20321,CVE-2021-38208,CVE-2021-4154,CVE-2022-0812,CVE-2022-1158,CVE-2022-1280,CVE-2022-1353,CVE-2022-1419,CVE-2022-1516,CVE-2022-28356,CVE-2022-28748,CVE-2022-28893,CVE-2022-29156
JIRA References: SLE-13208,SLE-13513,SLE-15172,SLE-15175,SLE-18234,SLE-8449
Sources used:
openSUSE Leap 15.4 (src):    dtb-aarch64-5.3.18-150300.59.68.1, kernel-preempt-5.3.18-150300.59.68.1
openSUSE Leap 15.3 (src):    dtb-aarch64-5.3.18-150300.59.68.1, kernel-64kb-5.3.18-150300.59.68.1, kernel-debug-5.3.18-150300.59.68.1, kernel-default-5.3.18-150300.59.68.1, kernel-default-base-5.3.18-150300.59.68.1.150300.18.41.3, kernel-docs-5.3.18-150300.59.68.1, kernel-kvmsmall-5.3.18-150300.59.68.1, kernel-obs-build-5.3.18-150300.59.68.1, kernel-obs-qa-5.3.18-150300.59.68.1, kernel-preempt-5.3.18-150300.59.68.1, kernel-source-5.3.18-150300.59.68.1, kernel-syms-5.3.18-150300.59.68.1, kernel-zfcpdump-5.3.18-150300.59.68.1
SUSE Linux Enterprise Workstation Extension 15-SP3 (src):    kernel-default-5.3.18-150300.59.68.1, kernel-preempt-5.3.18-150300.59.68.1
SUSE Linux Enterprise Module for Live Patching 15-SP3 (src):    kernel-default-5.3.18-150300.59.68.1, kernel-livepatch-SLE15-SP3_Update_18-1-150300.7.5.1
SUSE Linux Enterprise Module for Legacy Software 15-SP3 (src):    kernel-default-5.3.18-150300.59.68.1
SUSE Linux Enterprise Module for Development Tools 15-SP3 (src):    kernel-docs-5.3.18-150300.59.68.1, kernel-obs-build-5.3.18-150300.59.68.1, kernel-preempt-5.3.18-150300.59.68.1, kernel-source-5.3.18-150300.59.68.1, kernel-syms-5.3.18-150300.59.68.1
SUSE Linux Enterprise Module for Basesystem 15-SP3 (src):    kernel-64kb-5.3.18-150300.59.68.1, kernel-default-5.3.18-150300.59.68.1, kernel-default-base-5.3.18-150300.59.68.1.150300.18.41.3, kernel-preempt-5.3.18-150300.59.68.1, kernel-source-5.3.18-150300.59.68.1, kernel-zfcpdump-5.3.18-150300.59.68.1
SUSE Linux Enterprise Micro 5.2 (src):    kernel-default-5.3.18-150300.59.68.1, kernel-default-base-5.3.18-150300.59.68.1.150300.18.41.3
SUSE Linux Enterprise Micro 5.1 (src):    kernel-default-5.3.18-150300.59.68.1, kernel-default-base-5.3.18-150300.59.68.1.150300.18.41.3
SUSE Linux Enterprise High Availability 15-SP3 (src):    kernel-default-5.3.18-150300.59.68.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

Comment 36 Axel Braun 2022-05-16 20:38:55 UTC

Update to Kernel 5.3.18-150300.59.68 solves the problem on my RP4.
Thanks to everybody involved!

Comment 37 Ivan Ivanov 2022-05-19 13:09:02 UTC

And closing, again.

Comment 38 Ivan Ivanov 2022-05-20 14:35:10 UTC

*** Bug 1199764 has been marked as a duplicate of this bug. ***

Comment 40 Swamp Workflow Management 2022-06-14 22:32:08 UTC

SUSE-SU-2022:2079-1: An update that solves 15 vulnerabilities, contains two features and has 36 fixes is now available.

Category: security (important)
Bug References: 1055117,1061840,1065729,1103269,1118212,1152472,1152489,1153274,1154353,1156395,1158266,1167773,1176447,1178134,1180100,1183405,1188885,1195612,1195651,1195826,1196426,1196478,1196570,1196840,1197446,1197472,1197601,1197675,1198438,1198534,1198577,1198971,1198989,1199035,1199052,1199063,1199114,1199314,1199505,1199507,1199564,1199626,1199631,1199650,1199670,1199839,1200019,1200045,1200046,1200192,1200216
CVE References: CVE-2019-19377,CVE-2021-33061,CVE-2022-0168,CVE-2022-1184,CVE-2022-1652,CVE-2022-1729,CVE-2022-1972,CVE-2022-20008,CVE-2022-21123,CVE-2022-21125,CVE-2022-21127,CVE-2022-21166,CVE-2022-21180,CVE-2022-24448,CVE-2022-30594
JIRA References: SLE-13521,SLE-16387
Sources used:
openSUSE Leap 15.3 (src):    kernel-azure-5.3.18-150300.38.59.1, kernel-source-azure-5.3.18-150300.38.59.1, kernel-syms-azure-5.3.18-150300.38.59.1
SUSE Linux Enterprise Module for Public Cloud 15-SP3 (src):    kernel-azure-5.3.18-150300.38.59.1, kernel-source-azure-5.3.18-150300.38.59.1, kernel-syms-azure-5.3.18-150300.38.59.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.