Bug 625041

Summary: KVM/Xorg/fbdev: X segfaults on logout
Product: [openSUSE] openSUSE 11.3 Reporter: Bernhard Wiedemann <novellbmw>
Component: X.OrgAssignee: Michal Srb <msrb>
Status: RESOLVED FIXED QA Contact: E-mail List <xorg-maintainer-bugs>
Severity: Normal    
Priority: P5 - None CC: agraf, bwiedemann, msrb, sndirsch
Version: Final   
Target Milestone: ---   
Hardware: All   
OS: openSUSE 11.3   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: debug output with debuginfo and debugsource packages installed on openSUSE-11.1 (as 11.3 did not resolve addrs to symbols)
Xorg log with full debug+backtrace
0001-To-access-i-o-ports-above-0x3ff-we-need-to-be-in-iop.patch
fbdev Xorg log with full debug+backtrace
xf86-input-vmmouse-iopl-driver.diff
vmmouse_drv.so
fbdev Xorg log with full debug+backtrace
Enable hardware access during vmmouse preinit.

Description Bernhard Wiedemann 2010-07-23 11:01:04 UTC
Created attachment 378001 [details]
debug output with debuginfo and debugsource packages installed on openSUSE-11.1 (as 11.3 did not resolve addrs to symbols)

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.1.11) Gecko/20100714 SUSE/2.0.6-2.1 SeaMonkey/2.0.6

openSUSE in KVM uses the cirrus driver by default, but can be run with the vesa and fbdev drivers, too.
Activating fbdev in openSUSE-11.3 and 11.1 (but not 11.0 and 11.2, which do not use vmmouse) lets the X-server segfault in vmmouse_drv.so

Reproducible: Always

Steps to Reproduce:
0. have installed openSUSE's current kvm-0.12.3
1. qemu-kvm -m 1024 -cdrom openSUSE-11.3-KDE4-LiveCD-x86_64.iso
2. as root set Driver to fbdev via /etc/X11/xorg.conf.d/50-d*
3. rcxdm stop
4. startx
Actual Results:  
X-Server segfaults in vmmouse_drv.so VMMouseProto_SendCmd

Expected Results:  
should not segfault, but start KDE
Comment 1 Stefan Dirsch 2010-08-17 03:33:43 UTC
> 2. as root set Driver to fbdev via /etc/X11/xorg.conf.d/50-d*

Why are you doing this?
Comment 2 Bernhard Wiedemann 2010-08-17 04:24:09 UTC
This line refers to /etc/X11/xorg.conf.d/50-device.conf which I thought to be the standard place to add <Driver "fbdev"> 
to choose that driver, in case it is more stable than vesa or cirrus.

fbdev does actually work after disabling /etc/X11/xorg.conf.d/50-vmmouse.conf
even though vmmouse works with cirrus.
Comment 3 Stefan Dirsch 2010-08-17 07:29:04 UTC
Sure, but isn't using cirrus faster than vesa/fbdev? Also now you see a crash in vmmouse when using fbdev.
Comment 4 Matthias Hopf 2010-08-19 13:22:51 UTC
Yes, cirrus is faster. Still, this indicates there's a bug in the vmmouse module that could also surface under different conditions.
Comment 5 Matthias Hopf 2010-08-19 17:08:29 UTC
This is related to bug 604966. An equivalent call to iopl(3) as in attachment 366092 [details] works in this case as well.

Alex, do you know why qemu in *some* cases behaves differently here?

If you don't, we probably should use a different patch which only sets iopl(3) while sending the command, and reverts this effect afterwards directly afterwards. Still, I don't see where in vmmouse_proto.c the io port is actually accessed...

Also I don't know how to detect the current iopl level, so you can re-set it afterwards again.
Comment 7 Stefan Dirsch 2010-08-30 13:41:37 UTC
agraf proposed to give SLE11-SP1 kernel + appropriate kvm kernelmodule package
a try on the 11.1 host system.
Comment 8 Matthias Hopf 2010-09-15 15:15:19 UTC
No difference with the SP1 kernel, though it is much more stable otherwise.

I would suggest adding the call again in the vmmouse driver.
Comment 9 Stefan Dirsch 2010-11-05 22:44:14 UTC
Any improvements with latest updates from obs://X11:XOrg and obs://Kernel:HEAD?

zypper ar -f \
  http://download.opensuse.org/repositories/Kernel:/HEAD/openSUSE_11.3/ \
  Kernel:HEAD
zypper ar -f \
  http://download.opensuse.org/repositories/X11:/XOrg/openSUSE_11.3/ \
  X11:XOrg
zypper mr -p 90 Kernel:HEAD
zypper mr -p 90 X11:XOrg
zypper ref -r Kernel:HEAD
zypper ref -r X11:XOrg
zypper dup -r Kernel:HEAD
zypper dup -r X11:XOrg
zypper in xorg-x11-server-debuginfo xorg-x11-server-debugsource \
          xorg-x11-driver-video-debuginfo xorg-x11-driver-video-debugsource \
          xorg-x11-driver-input-debuginfo xorg-x11-driver-input-debugsource \
          libpixman-1-0-debuginfo libpixman-1-0-debugsource \
          libpciaccess0-debuginfo libpciaccess0-debugsource
Comment 10 Bernhard Wiedemann 2010-11-06 14:49:30 UTC
Created attachment 398872 [details]
Xorg log with full debug+backtrace

This is from a crash on a current Factory with above repos+rpms added within a kvm-0.12.4 on my Debian/lenny host (can only try SLES11-SP1 on Monday)
Comment 11 Stefan Dirsch 2010-11-06 15:09:28 UTC
Ok. Thanks for the update.
Comment 12 Stefan Dirsch 2010-11-06 15:43:25 UTC
(In reply to comment #5)
> This is related to bug 604966. An equivalent call to iopl(3) as in attachment
> 366092 works in this case as well.
> 
> Alex, do you know why qemu in *some* cases behaves differently here?
> 
> If you don't, 

Seems so. :-(

> we probably should use a different patch which only sets iopl(3)
> while sending the command, and reverts this effect afterwards directly
> afterwards. 

That would be in shared/vmmouse_proto.c:VMMouseProto_SendCmd() right before/after calling VMMouseProtoInOut(). Makes perfectly sense to me.

> Still, I don't see where in vmmouse_proto.c the io port is  actually 
> accessed...

Obviously that's in VMMouseProtoInOut(...) 
  __asm__ __volatile__(

Or what do you mean exactly?

> Also I don't know how to detect the current iopl level, so you can re-set it
> afterwards again.

Me not either. Can we assume that the Xserver runs as a "normal process"?

# man iopl
[...]
The I/O privilege level for a normal process is 0.
[...]

Then it would be easy, but I doubt that. Maybe it's Level 1, maybe it's level 2.
Comment 14 Alexander Graf 2010-11-07 01:05:14 UTC
Sorry, I talked to Matthias in person a while back about this. Apparently information flow stopped there :).

I'm not aware of any way to detect the current io privilege level. I'm missing the requirement for it though. Why would you need to change it back when setting it to 3? Just let X access any io port when we have the vmmouse drive in use.

Btw, the levels seem to indicate the rings in the x86 cpu. Ring 0 is kernel space, ring 3 is user space. Rings 1 and 2 are unused on i386 and are non-existent on x86_64.
Comment 16 Stefan Dirsch 2010-11-07 05:37:48 UTC
Thanks for the input, Alex. I'm just wondering, whether it's a good idea to switch the Xserver in between to user space. I still would expect problems when trying to do that. Therefore it might be better to switch back to kernel space.
Jiri Kosina sent me a code snippet how to detect the current I/O privilege level.
Comment 17 Stefan Dirsch 2010-11-07 12:02:03 UTC
Created attachment 398903 [details]
0001-To-access-i-o-ports-above-0x3ff-we-need-to-be-in-iop.patch

Proposal.
Comment 18 Stefan Dirsch 2010-11-07 12:30:22 UTC
Patch is now included in X11:XOrg/xorg-x11-driver-input. Package has been rebuilt and is now available via

  http://download.opensuse.org/repositories/X11:/XOrg/openSUSE_11.3/

RPM changelog entry:

Sun Nov  7 12:05:53 UTC 2010 - sndirsch@novell.com
- xf86-input-vmmouse-iopl-driver.diff
  * switch to I/O privilege level 3 also for the driver itself
    (bnc #625041)

Please install also the -debug* packages. In any case please attach the new
logfile, since I've added some debug messages and would like to know how often
the I/O privilege level is changed.
Comment 19 Bernhard Wiedemann 2010-11-07 15:33:40 UTC
Created attachment 398904 [details]
fbdev Xorg log with full debug+backtrace

This works much better now, running KDE4, konsole and glxgears without problems... it only crashed on logout.
Comment 20 Bernhard Wiedemann 2010-11-07 15:34:35 UTC
forgot to clear NEEDINFO flag.
Comment 21 Stefan Dirsch 2010-11-07 18:01:17 UTC
(In reply to comment #19)
> Created an attachment (id=398904) [details]
> fbdev Xorg log with full debug+backtrace

Well, apparently the new code I've added didn't get active at all. Either you didn't update the xorg-x11-driver-input package or I patched dead code. Also there is no crash any longer in vmmouse_drv.

> This works much better now, running KDE4, konsole and glxgears without
> problems... it only crashed on logout.

That's a complete different crash in glx module it seems.
Comment 22 Stefan Dirsch 2010-11-07 18:06:42 UTC
Bernhard, could you please verify that the following is part of the output when
running 'rpm --changelog -q xorg-x11-driver-input'?

Sun Nov  7 12:05:53 UTC 2010 - sndirsch@novell.com
- xf86-input-vmmouse-iopl-driver.diff
  * switch to I/O privilege level 3 also for the driver itself
    (bnc #625041)
Comment 23 Stefan Dirsch 2010-11-07 18:14:15 UTC
Argh. VMwareLog(()) is only useful, if VMX86_DEVEL is defined, which apparently is not. I'm going to replace it with xf86Msg(X_INFO, ...).
Comment 24 Stefan Dirsch 2010-11-07 18:31:32 UTC
(In reply to comment #23)
> Argh. VMwareLog(()) is only useful, if VMX86_DEVEL is defined, which apparently
> is not. I'm going to replace it with xf86Msg(X_INFO, ...).

... which I can't that easily since the code is shared between the driver and the standalone tool. :-(
Comment 25 Bernhard Wiedemann 2010-11-07 19:24:26 UTC
I had used your lines from comment #9 to update. The changelog does indeed have your "Nov 7 2010" entry, so it was just the extra logging not working. The bugfix for the original fbdev+vmmouse issue is good.

Also added your update script to http://en.opensuse.org/openSUSE:Bugreport_X since it is pretty useful for debugging and testing.
Comment 26 Stefan Dirsch 2010-11-07 21:21:25 UTC
Created attachment 398908 [details]
xf86-input-vmmouse-iopl-driver.diff

Use fprintf(...) instead of VMwareLog((...)) for debug messages. Possibly these debug messages will occur in /var/log/xdm.errors or /var/log/kdm.log instead of /var/log/Xorg.0.log.
Comment 27 Stefan Dirsch 2010-11-07 22:20:40 UTC
Looks like fprintf() also goes to /dev/null when used in Xserver (modules). :-(
Comment 28 Stefan Dirsch 2010-11-08 00:04:21 UTC
Created attachment 398913 [details]
vmmouse_drv.so

Please copy this file (vmmouse_drv.so) manually to /usr/lib/xorg/modules/drivers/. Now the debug messages should occur in the X logfile like

Current I/O privilege level: ...
Switching to I/O privilege level 3.
Switching back to previous I/O privilege level ...

Please attach the new logfile.
Comment 29 Bernhard Wiedemann 2010-11-08 05:05:20 UTC
Created attachment 398920 [details]
fbdev Xorg log with full debug+backtrace

There are 27614 entries about Switching in a 90-second test, often only one millisecond apart.
Comment 30 Stefan Dirsch 2010-11-08 07:07:06 UTC
Thanks a lot. That's what I was afraid of. I guess we don't want these switches. At least not that often. AFAIK they are rather CPU expensive.
Comment 31 Stefan Dirsch 2010-11-08 15:20:47 UTC
Nevetheless I've now submitrequested an updated xorg-x11-driver-input for factory, since that's still better than a crashing Xserver due to the vmmouse usage.

-------------------------------------------------------------------
Mon Nov  8 14:57:59 UTC 2010 - sndirsch@novell.com

- merged xf86-input-vmmouse-iopl.diff/
  xf86-input-vmmouse-iopl-driver.diff and removed fprintfs

-------------------------------------------------------------------
Sun Nov  7 21:14:34 UTC 2010 - sndirsch@novell.com

- xf86-input-vmmouse-iopl-driver.diff
  * Use fprintf(...) instead of VMwareLog((...)) for debug messages. 

-------------------------------------------------------------------
Sun Nov  7 12:05:53 UTC 2010 - sndirsch@novell.com

- xf86-input-vmmouse-iopl-driver.diff
  * switch to I/O privilege level 3 also for the driver itself
    (bnc #625041)
Comment 32 Stefan Dirsch 2010-11-27 12:34:44 UTC
Ok. Let's live with that approach. Matthias told me that these iopl() calls aren't that expensive as I thought first.
Comment 33 Michal Srb 2012-05-03 14:53:14 UTC
Reopening because of bug #743401.
Comment 34 Michal Srb 2012-05-03 14:59:02 UTC
Created attachment 489411 [details]
Enable hardware access during vmmouse preinit.

This patch enables hardware access using Xserver's xf86EnableIO function. It is the same approach as in xf86-input-keyboard.

This should be used instead of xf86-input-vmmouse-iopl.diff, results:
* The iopl is no longer called before and after every outl.
* It won't accidentally switch to iopl(0) because of wrong EFLAGS value in XEN
  in VMWARE. (bug #743401).
Comment 35 Michal Srb 2012-11-06 15:21:55 UTC
Closing as resolved because of bug #743401.
Comment 36 Bernhard Wiedemann 2013-08-30 15:00:13 UTC
This is an autogenerated message for OBS integration:
This bug (625041) was mentioned in
https://build.opensuse.org/request/show/196893 Factory / xf86-input-vmmouse