Bug 406638

Summary: persistent segfaults in major services
Product: [openSUSE] openSUSE 11.0 Reporter: Pavel Koshevoy <paul>
Component: OtherAssignee: Holger Macht <hmacht>
Status: RESOLVED DUPLICATE QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: on, paul, varkoly
Version: Final   
Target Milestone: ---   
Hardware: x86-64   
OS: openSUSE 11.0   
Whiteboard:
Found By: Customer Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: lshal output
lspci output
dmesg output after a fresh boot
lsmod output
lsdev output

Description Pavel Koshevoy 2008-07-06 18:01:25 UTC
Clean install of openSUSE 11.0 x86_64 on ASUS P5W DH Deluxe, with 4GB DDR2-800 RAM and Core2 Extreme x6800

When I use YaST to install a new package, or do some other configuration, major services running on this machine crash. This often happens when restarting hald or dbus daemon. The services have to be restarted manually. This is a problem, because it requires continuous keep monitoring to make sure that postfix, apache and samba services haven't died.

The problem did not exist on openSUSE 10.3 which was running on this computer until July 4th.

The following is an excerpt from /var/log/messages generated by grepping for segfault:

Jul  5 19:30:01 homestead kernel: cron[17382]: segfault at f00 ip 7f622b7f7d9b sp 7fff33a17c80 error 4 in ld-2.8.so[7f622b7ef000+1d000]
Jul  5 19:45:01 homestead kernel: cron[17403]: segfault at f00 ip 7f622b7f7d9b sp 7fff33a17c80 error 4 in ld-2.8.so[7f622b7ef000+1d000]
Jul  5 20:00:01 homestead kernel: cron[17405]: segfault at f00 ip 7f622b7f7d9b sp 7fff33a17c80 error 4 in ld-2.8.so[7f622b7ef000+1d000]
Jul  5 20:15:01 homestead kernel: cron[17406]: segfault at f00 ip 7f622b7f7d9b sp 7fff33a17c80 error 4 in ld-2.8.so[7f622b7ef000+1d000]
Jul  5 20:30:01 homestead kernel: cron[17469]: segfault at f00 ip 7f622b7f7d9b sp 7fff33a17c80 error 4 in ld-2.8.so[7f622b7ef000+1d000]
Jul  5 20:45:01 homestead kernel: cron[17705]: segfault at f00 ip 7f622b7f7d9b sp 7fff33a17c80 error 4 in ld-2.8.so[7f622b7ef000+1d000]
Jul  5 21:14:09 homestead kernel: powersaved[11432]: segfault at 23d5b4 ip 7f7a8952b0d6 sp 7fff928ef0c0 error 4 in libdbus-1.so.3.4.0[7f7a8951f000+3c000]
Jul  5 21:14:09 homestead kernel: hald[12503]: segfault at 23d5b4 ip 7fb83954e0d6 sp 7fff420c1a40 error 4 in libdbus-1.so.3.4.0[7fb839542000+3c000]
Jul  5 21:14:09 homestead kernel: console-kit-dae[9915]: segfault at 23d5b4 ip 7f5ab74930d6 sp 7fffbfd1c7c0 error 4 in libdbus-1.so.3.4.0[7f5ab7487000+3c000]
Jul  5 21:14:09 homestead kernel: hald-addon-inpu[12522]: segfault at 23d5b4 ip 7f5a718a60d6 sp 7fff7a125ab0 error 4<6>hald-addon-cpuf[12524]: segfault at 23d5b4
 ip 7fec9cb270d6 sp 7fffa53a8170 error 4 in libdbus-1.so.3.4.0[7fec9cb1b000+3c000] in libdbus-1.so.3.4.0[7f5a7189a000+3c000]
Jul  5 21:14:09 homestead kernel: hald-runner[12504]: segfault at 23d5b4 ip 7fac24a270d6 sp 7fff2d35aea0 error 4 in libdbus-1.so.3.4.0[7fac24a1b000+3c000]
Jul  5 21:14:32 homestead kernel: resmgrd[12614]: segfault at 110e ip 110e sp 7fffcc4c7b28 error 14 in resmgrd[400000+9000]
Jul  5 21:15:01 homestead kernel: cron[19456]: segfault at f00 ip 7f86d7912d9b sp 7fffdfb32500 error 4 in ld-2.8.so[7f86d790a000+1d000]
Jul  5 21:16:31 homestead kernel: imapd[17446]: segfault at 22b6 ip 22b6 sp 7fffce3ca028 error 14 in imapd[400000+fd000]
Jul  5 22:08:35 homestead kernel: sshd[17477]: segfault at 22b6 ip 22b6 sp 7fff8861b088 error 14 in zero (deleted)[7f627ac15000+140000]
Jul  5 22:08:35 homestead kernel: su[17708]: segfault at 22b6 ip 22b6 sp 7fff8d1abe38 error 14 in pam_deny.so[7f5b82562000+1000]
Jul  5 22:46:03 homestead kernel: freshclam[10907]: segfault at 33d6 ip 33d6 sp 7fff676fbbf8 error 14 in freshclam[400000+14000]
Jul  6 11:27:10 homestead kernel: hald[2283]: segfault at 23d5b4 ip 7fcba97090d6 sp 7fffb227ed20 error 4 in libdbus-1.so.3.4.0[7fcba96fd000+3c000]
Jul  6 11:27:10 homestead kernel: hald-runner[2345]: segfault at 23d5b4 ip 7f16ac79a0d6 sp 7fffb50ce890 error 4 in libdbus-1.so.3.4.0[7f16ac78e000+3c000]
Jul  6 11:27:10 homestead kernel: powersaved[3718]: segfault at 23d5b4 ip 7f52a4eaa0d6 sp 7fffae26ef20 error 4 in libdbus-1.so.3.4.0[7f52a4e9e000+3c000]
Jul  6 11:27:10 homestead kernel: hald-addon-cpuf[2449]: segfault at 23d5b4 ip 7f76fefae0d6 sp 7fff0782ea10 error 4 in libdbus-1.so.3.4.0[7f76fefa2000+3c000]
Jul  6 11:27:10 homestead kernel: hald-addon-stor[2482]: segfault at 23d5b4 ip 7f03141cf0d6 sp 7fff1ca4e630 error 4 in libdbus-1.so.3.4.0[7f03141c3000+3c000]
Jul  6 11:27:10 homestead kernel: hald-addon-stor[2485]: segfault at 23d5b4 ip 7f10e44d80d6 sp 7fffecd57d80 error 4 in libdbus-1.so.3.4.0[7f10e44cc000+3c000]
Jul  6 11:27:10 homestead kernel: console-kit-dae[2280]: segfault at 23d5b4 ip 7ff1af5b00d6 sp 7fffb7e38000 error 4 in libdbus-1.so.3.4.0[7ff1af5a4000+3c000]
Jul  6 11:29:30 homestead kernel: resmgrd[2231]: segfault at 110e ip 110e sp 7fff12096058 error 14 in resmgrd[400000+9000]
Jul  6 11:30:01 homestead kernel: cron[6688]: segfault at f00 ip 7f7c7903cd9b sp 7fff8125bbd0 error 4 in ld-2.8.so[7f7c79034000+1d000]
Jul  6 11:45:01 homestead kernel: dbus-daemon[2125]: segfault at 110e ip 110e sp 7fff96ee2608 error 14 in libnss_files-2.8.so[7f248dec3000+a000]
Comment 1 Pavel Koshevoy 2008-07-06 18:23:05 UTC
Created attachment 226152 [details]
lshal output
Comment 2 Pavel Koshevoy 2008-07-06 18:23:39 UTC
Created attachment 226153 [details]
lspci output
Comment 3 Pavel Koshevoy 2008-07-06 18:25:01 UTC
Created attachment 226154 [details]
dmesg output after a fresh boot
Comment 4 Pavel Koshevoy 2008-07-06 18:26:30 UTC
Created attachment 226155 [details]
lsmod output
Comment 5 Pavel Koshevoy 2008-07-06 18:27:40 UTC
Created attachment 226156 [details]
lsdev output
Comment 6 Pavel Koshevoy 2008-07-12 15:50:06 UTC
It's still happening -- every time when I use YaST to install a package or run SuSEconfig -module postfix


Jul  5 21:14:09 homestead kernel: hald-runner[12504]: segfault at 23d5b4 ip 7fac24a270d6 sp 7fff2d35aea0 error 4 in libdbus-1.so.3.4.0[7fac24a1b000+3c000]
Jul  5 21:14:32 homestead kernel: resmgrd[12614]: segfault at 110e ip 110e sp 7fffcc4c7b28 error 14 in resmgrd[400000+9000]
Jul  5 21:15:01 homestead kernel: cron[19456]: segfault at f00 ip 7f86d7912d9b sp 7fffdfb32500 error 4 in ld-2.8.so[7f86d790a000+1d000]
Jul  5 21:16:31 homestead kernel: imapd[17446]: segfault at 22b6 ip 22b6 sp 7fffce3ca028 error 14 in imapd[400000+fd000]
Jul  5 22:08:35 homestead kernel: sshd[17477]: segfault at 22b6 ip 22b6 sp 7fff8861b088 error 14 in zero (deleted)[7f627ac15000+140000]
Jul  5 22:08:35 homestead kernel: su[17708]: segfault at 22b6 ip 22b6 sp 7fff8d1abe38 error 14 in pam_deny.so[7f5b82562000+1000]
Jul  5 22:46:03 homestead kernel: freshclam[10907]: segfault at 33d6 ip 33d6 sp 7fff676fbbf8 error 14 in freshclam[400000+14000]
Jul  6 11:27:10 homestead kernel: hald[2283]: segfault at 23d5b4 ip 7fcba97090d6 sp 7fffb227ed20 error 4 in libdbus-1.so.3.4.0[7fcba96fd000+3c000]
Jul  6 11:27:10 homestead kernel: hald-runner[2345]: segfault at 23d5b4 ip 7f16ac79a0d6 sp 7fffb50ce890 error 4 in libdbus-1.so.3.4.0[7f16ac78e000+3c000]
Jul  6 11:27:10 homestead kernel: powersaved[3718]: segfault at 23d5b4 ip 7f52a4eaa0d6 sp 7fffae26ef20 error 4 in libdbus-1.so.3.4.0[7f52a4e9e000+3c000]
Jul  6 11:27:10 homestead kernel: hald-addon-cpuf[2449]: segfault at 23d5b4 ip 7f76fefae0d6 sp 7fff0782ea10 error 4 in libdbus-1.so.3.4.0[7f76fefa2000+3c000]
Jul  6 11:27:10 homestead kernel: hald-addon-stor[2482]: segfault at 23d5b4 ip 7f03141cf0d6 sp 7fff1ca4e630 error 4 in libdbus-1.so.3.4.0[7f03141c3000+3c000]
Jul  6 11:27:10 homestead kernel: hald-addon-stor[2485]: segfault at 23d5b4 ip 7f10e44d80d6 sp 7fffecd57d80 error 4 in libdbus-1.so.3.4.0[7f10e44cc000+3c000]
Jul  6 11:27:10 homestead kernel: console-kit-dae[2280]: segfault at 23d5b4 ip 7ff1af5b00d6 sp 7fffb7e38000 error 4 in libdbus-1.so.3.4.0[7ff1af5a4000+3c000]
Jul  6 11:29:30 homestead kernel: resmgrd[2231]: segfault at 110e ip 110e sp 7fff12096058 error 14 in resmgrd[400000+9000]
Jul  6 11:30:01 homestead kernel: cron[6688]: segfault at f00 ip 7f7c7903cd9b sp 7fff8125bbd0 error 4 in ld-2.8.so[7f7c79034000+1d000]
Jul  6 11:45:01 homestead kernel: dbus-daemon[2125]: segfault at 110e ip 110e sp 7fff96ee2608 error 14 in libnss_files-2.8.so[7f248dec3000+a000]
Jul  6 12:06:13 homestead kernel: su[5317]: segfault at 22b6 ip 22b6 sp 7fff0a1818a8 error 14 in pam_deny.so[7f05ff53a000+1000]
Jul  6 12:06:13 homestead kernel: su[5168]: segfault at 22b6 ip 22b6 sp 7fff41f0d808 error 14 in pam_deny.so[7f9d372c3000+1000]
Jul  6 12:06:32 homestead kernel: kdm[4820]: segfault at 6c26 ip 6c26 sp 7fff25d94328 error 14 in kdm[400000+25000]
Jul  8 20:03:00 homestead kernel: hald[2348]: segfault at 23d5b4 ip 7f4d6e6890d6 sp 7fff771fed00 error 4 in libdbus-1.so.3.4.0[7f4d6e67d000+3c000]
Jul  8 20:03:00 homestead kernel: powersaved[3718]: segfault at 23d5b4 ip 7f459a63a0d6 sp 7fffa39fe8f0 error 4 in libdbus-1.so.3.4.0[7f459a62e000+3c000]
Jul  8 20:03:00 homestead kernel: hald-runner[2349]: segfault at 26c8 ip 7f8591e0ed9b sp 7fff9a023530 error 4 in ld-2.8.so[7f8591e06000+1d000]
Jul  8 20:03:00 homestead kernel: console-kit-dae[2272]: segfault at 23d5b4 ip 7fcee8cac0d6 sp 7ffff1534430 error 4 in libdbus-1.so.3.4.0[7fcee8ca0000+3c000]
Jul  8 20:03:00 homestead kernel: hald-addon-inpu[2438]: segfault at 23d5b4 ip 7fbabadf40d6 sp 7fffc36753c0 error 4 in libdbus-1.so.3.4.0[7fbabade8000+3c000]
Jul  8 20:03:00 homestead kernel: hald-addon-cpuf[2447]: segfault at 23d5b4 ip 7f9a1cbe20d6 sp 7fff254632d0 error 4 in libdbus-1.so.3.4.0[7f9a1cbd6000+3c000]
Jul  8 20:03:24 homestead kernel: resmgrd[2183]: segfault at 110e ip 110e sp 7fffa8ff92f8 error 14 in resmgrd[400000+9000]
Jul  8 20:03:57 homestead kernel: su[7016]: segfault at 22b6 ip 22b6 sp 7fff14e23718 error 14 in pam_deny.so[7f150a1db000+1000]
Jul  8 20:05:34 homestead kernel: su[4830]: segfault at 22b6 ip 22b6 sp 7fff7fed7fc8 error 14 in libnss_files-2.8.so[7fff7528f000+a000]
Jul  8 20:05:34 homestead kernel: su[4865]: segfault at 22b6 ip 22b6 sp 7fff3f0b3878 error 14 in pam_deny.so[7f8d3446a000+1000]
Jul  8 20:05:39 homestead kernel: imapd[6322]: segfault at 22b6 ip 22b6 sp 7fff39ba73c8 error 14 in imapd[400000+fd000]
Jul  8 20:05:43 homestead kernel: kdm[3587]: segfault at 6c26 ip 6c26 sp 7fffec7d7ec8 error 14 in kdm[400000+25000]
Jul 11 07:55:48 homestead kernel: powersaved[3707]: segfault at 23d5b4 ip 7f93d753d0d6 sp 7fffe0901c90 error 4 in libdbus-1.so.3.4.0[7f93d7531000+3c000]
Jul 11 07:55:48 homestead kernel: hald[2333]: segfault at 23d5b4 ip 7fbaccdb10d6 sp 7fffd5926d30 error 4 in libdbus-1.so.3.4.0[7fbaccda5000+3c000]
Jul 11 07:55:48 homestead kernel: hald-addon-inpu[2464]: segfault at 23d5b4 ip 7f581c81f0d6 sp 7fff2509ff10 error 4<6>hald-addon-cpuf[2466]: segfault at 23d5b4 ip 7f403c2790d6 sp 7fff44af9000 error 4 in libdbus-1.so.3.4.0[7f403c26d000+3c000]
Jul 11 07:55:48 homestead kernel: hald-runner[2406]: segfault at 23d5b4 ip 7f9f2ba0a0d6 sp 7fff3433d440 error 4 in libdbus-1.so.3.4.0[7f9f2b9fe000+3c000]
Jul 11 07:55:48 homestead kernel: hald-addon-stor[2512]: segfault at 23d5b4 ip 7fa02b8790d6 sp 7fff340f8120 error 4 in libdbus-1.so.3.4.0[7fa02b86d000+3c000]
Jul 11 07:55:48 homestead kernel: console-kit-dae[2332]: segfault at 23d5b4 ip 7f125b6100d6 sp 7fff63e97ca0 error 4 in libdbus-1.so.3.4.0[7f125b604000+3c000]
Jul 11 07:55:48 homestead kernel: hald-addon-stor[2509]: segfault at 23d5b4 ip 7f39e49930d6 sp 7fffed213df0 error 4 in libdbus-1.so.3.4.0[7f39e4987000+3c000]
Jul 11 07:56:04 homestead kernel: su[32726]: segfault at 22b6 ip 22b6 sp 7ffff755b208 error 14 in pam_deny.so[7f56ec911000+1000]
Jul 11 07:56:06 homestead kernel: su[4892]: segfault at 22b6 ip 22b6 sp 7ffffd0f48b8 error 14 in pam_deny.so[7fd7f24ab000+1000]
Jul 11 07:56:06 homestead kernel: su[4927]: segfault at 22b6 ip 22b6 sp 7fff049dc1a8 error 14 in pam_deny.so[7fb9f9d94000+1000]
Jul 11 07:56:06 homestead kernel: kdm[3593]: segfault at d68 ip 7f33bfd07d9b sp 7fffc7f1bad0 error 4 in ld-2.8.so[7f33bfcff000+1d000]
Comment 9 Michal Zugec 2008-07-17 06:41:50 UTC
Pavel, we can't reproduce this problem. Seems to be hardware issue. Could you try a memtest?
Comment 10 Pavel Koshevoy 2008-07-17 19:46:47 UTC
OK, I'll try tonight when the server is not used much. I doubt it's the memory though -- I've had the same memory since September 2006 (4 sticks of CORSAIR XMS2 1GB).  Also, I don't see random crashes -- it crashes consistently after installing a new rpm with YaST, when it runs SuSEconfig. It crashes yesterday when I installed boost-devel package.
Comment 11 Pavel Koshevoy 2008-07-18 13:38:33 UTC
I ran the memtest for 6.5 hrs last night.  It did 6 passes and didn't find any errors.
Comment 13 Ørnulf Nielsen 2008-07-22 12:09:48 UTC
I've this bug on a machine as well (IBM 335 8676-21G). I noticed it hangs when starting powersaved (logs show segfaults similar to Pavel's).

Noticed in the logs:
powersaved[3539]: WARNING (CpufreqManagement:52) No capability cpufreq_control

After removing powersaved (insserv -r /etc/init.d/powersaved) from startup, the problem does not occur.
Comment 15 Holger Macht 2008-07-22 14:53:33 UTC
Please boot with CPUFREQ=off. Still happening?
Comment 16 Pavel Koshevoy 2008-07-23 00:48:26 UTC
Booting with CPUFREQ=off did not help.  I've changed /etc/sysconfig/postfix to disable chroot jail (set POSTFIX_CHROOT="no" and POSTFIX_UPDATE_CHROOT_JAIL="no"), and commented out a line if /etc/fstab

#/var/run/sasl2       /var/spool/postfix/var/run/sasl2 none rw,bind 0 0

I've rebooted and so far I can't reproduce the segfaults.  
Is the problem with chroot?
Comment 17 Ørnulf Nielsen 2008-07-23 08:25:17 UTC
Disregard comment #13 - the bug appeared again :-/

I tried disabling postfix chroot (se comment #16) - no segfaults yet
Comment 18 Peter Varkoly 2008-07-31 09:06:17 UTC
The chroot works fine. I think the problem were your fstab entry. Please let the SuSE-Postfix scripts to build the chroot environment. 
Comment 19 Pavel Koshevoy 2008-07-31 23:58:40 UTC
(In reply to comment #18 from Peter Varkoly)
> The chroot works fine. I think the problem were your fstab entry. Please let
> the SuSE-Postfix scripts to build the chroot environment. 
> 

I re-enabled chroot jail for postfix (without enabling my changes to /etc/fstab for sasl authentication), and reran SuSEconfig -module postfix. Immediately several applications crashed, including Thunderbird and Gimp. See the following dmesg excerpt:

gimp[19019] general protection ip:7f6433b1e6f4 sp:7fff3f78ce48 error:0 in libdbus-1.so.3.4.0[7f6433afd000+3c000]
yauap[4900]: segfault at 23d5b4 ip 7ffa80d140d6 sp 7fff8a6ada10 error 4 in libdbus-1.so.3.4.0[7ffa80d08000+3c000]
hald-addon-stor[2487] general protection ip:7f540a9716f4 sp:7fff131dcdf8 error:0 in libdbus-1.so.3.4.0[7f540a950000+3c000]
hald[2354]: segfault at 23d5b4 ip 7f75122660d6 sp 7fff1addadb0 error 4 in libdbus-1.so.3.4.0[7f751225a000+3c000]
powersaved[3658]: segfault at 23d5b4 ip 7fb3deb110d6 sp 7fffe7ed6c40 error 4 in libdbus-1.so.3.4.0[7fb3deb05000+3c000]
hald-runner[2355]: segfault at 23d5b4 ip 7f7e1c6920d6 sp 7fff24fc4e50 error 4 in libdbus-1.so.3.4.0[7f7e1c686000+3c000]
hald-addon-stor[2490]: segfault at 23d5b4 ip 7fbc9c0a00d6 sp 7fffa491f950 error 4 in libdbus-1.so.3.4.0[7fbc9c094000+3c000]
hald-addon-cpuf[2450]: segfault at 23d5b4 ip 7f4913e8c0d6 sp 7fff1c70b580 error 4 in libdbus-1.so.3.4.0[7f4913e80000+3c000]
hald-addon-inpu[2439]: segfault at 23d5b4 ip 7f7c866050d6 sp 7fff8ee85ea0 error 4 in libdbus-1.so.3.4.0[7f7c865f9000+3c000]
avahi-daemon[3503] general protection ip:7f177f9bd6f4 sp:7fff8886d708 error:0 in libdbus-1.so.3.4.0[7f177f99c000+3c000]
printk: 1 messages suppressed.
imapd[22342]: segfault at 22b6 ip 22b6 sp 7ffffc933558 error 14 in imapd[400000+fd000]
Comment 20 Peter Varkoly 2008-08-01 15:02:29 UTC
In ftp://ftp.suse.com/pub/people/varkoly/postfix/11.0-<arch> you can find a postfix package which hase some fixes in the SuSE.postfix script creating the chroot jail. Please check this package.
Comment 21 Pavel Koshevoy 2008-08-02 14:55:25 UTC
(In reply to comment #20 from Peter Varkoly)
> In ftp://ftp.suse.com/pub/people/varkoly/postfix/11.0-<arch> you can find a
> postfix package which hase some fixes in the SuSE.postfix script creating the
> chroot jail. Please check this package.
> 

The download took an hour, 500 B/s -- what's wrong with ftp.suse.com? Anyway, I've updated postfix, enabled postfix chroot jail and ran SuSEconfig -module postfix -- no segfaults. I then restored my fstab to mount /var/run/sasl2 on  /var/spool/postfix/var/run/sasl2 -- no segfaults either. So far it's been running stable.

When will this fix become official and show up in the online update repositories?
Comment 22 Peter Varkoly 2008-08-02 20:05:26 UTC
Thank you for testing! I've already started the maintainace prozess for this bug.
Comment 23 Peter Varkoly 2008-08-02 20:09:31 UTC

*** This bug has been marked as a duplicate of bug 409104 ***