Bug 698125 - Xvfb crashes in miInitVisuals () with 1/4bit color depth.
Summary: Xvfb crashes in miInitVisuals () with 1/4bit color depth.
Status: RESOLVED UPSTREAM
Alias: None
Product: openSUSE 11.4
Classification: openSUSE
Component: X.Org (show other bugs)
Version: Final
Hardware: x86-64 openSUSE 11.4
: P3 - Medium : Critical with 1 vote (vote)
Target Milestone: ---
Assignee: Stefan Dirsch
QA Contact: E-mail List
URL: https://bugs.freedesktop.org/show_bug...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-06-04 12:57 UTC by Christopher Yeleighton
Modified: 2011-06-17 18:02 UTC (History)
0 users

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christopher Yeleighton 2011-06-04 12:57:30 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1

The X windows virtual frame buffer server crashes with the default depth preset to 2.

Reproducible: Always

Steps to Reproduce:
  1. { gdb Xvfb; }
  2. { run :1 -screen 0 10x10x2; }

Actual Results:  
  2.
Starting program: /usr/bin/Xvfb :1 -screen 0 10x10x2
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000543830 in miInitVisuals ()
(gdb) bt
#0  0x0000000000543830 in miInitVisuals ()
#1  0x00000000004fc502 in fbInitVisuals ()
#2  0x000000000041f2fb in fbFinishScreenInit ()
#3  0x000000000041f5d6 in fbScreenInit ()
#4  0x000000000041c87b in ?? ()
#5  0x000000000050daa8 in AddScreen ()
#6  0x000000000041dccf in InitOutput ()
#7  0x00000000004fc6ed in ?? ()
#8  0x00007ffff67c4bfd in __libc_start_main (main=0x4fc510, argc=5, ubp_av=0x7fffffffdc68, init=<value optimized out>, fini=<value optimized out>, 
    rtld_fini=<value optimized out>, stack_end=0x7fffffffdc58) at libc-start.c:226
#9  0x000000000041c189 in _start ()


Expected Results:  
  2,
The server should not crash.
Comment 1 Stefan Dirsch 2011-06-06 09:32:21 UTC
I'm wondering what's the use case for 2 bit color depth. 8, 15, 16, 24 bit sounds reasonable to me. Maybe also 1 and 4 bit (which is also broken I figured out meanwhile), but 2 bit?
Comment 2 Christopher Yeleighton 2011-06-06 12:03:57 UTC
I think I can disclose that we produce 4-color compressed hi-res raster images.
The problem is urgent, so I am going to look into it myself anyway; the trouble is the debugging information in Xvfb is too scarce.
Comment 3 Christopher Yeleighton 2011-06-06 22:07:09 UTC
using xorg-x11-server-7.6_1.9.3-15.18.4

Program received signal SIGSEGV, Segmentation fault.
0x0000000000543830 in miInitVisuals (visualp=0x7fffffffe478, depthp=0x7fffffffe470, nvisualp=0x7fffffff0001, ndepthp=<value optimized out>, rootDepthp=0x7fffffffe484, defaultVisp=0x7fffffffe480, sizes=128, bitsPerRGB=8, preferredVis=-1) at micmap.c:664
664       *defaultVisp = depth[i].vids[j];
#0  0x0000000000543830 in miInitVisuals (visualp=0x7fffffffe478, depthp=0x7fffffffe470, nvisualp=0x7fffffff0001, ndepthp=<value optimized out>, rootDepthp=0x7fffffffe484, defaultVisp=0x7fffffffe480, sizes=128, bitsPerRGB=8, preferredVis=-1) at micmap.c:664
#1  0x00000000004fc502 in fbInitVisuals (visualp=<value optimized out>, depthp=<value optimized out>, nvisualp=<value optimized out>, ndepthp=<value optimized out>, rootDepthp=<value optimized out>, defaultVisp=<value optimized out>, sizes=128, bitsPerRGB=8) at ../../fb/fbcmap_mi.c:122
#2  0x000000000041f2fb in fbFinishScreenInit (pScreen=0x7ebf30, pbits=0x7ec360, xsize=10, ysize=10, dpix=100, dpiy=100, width=12, bpp=8) at fbscreen.c:226
#3  0x000000000041f5d6 in fbScreenInit (pScreen=0x7ebf30, pbits=0x7ec360, xsize=10, ysize=10, dpix=100, dpiy=100, width=12, bpp=8) at fbscreen.c:283
#4  0x000000000041c87b in vfbScreenInit (index=<value optimized out>, pScreen=0x7ebf30, argc=<value optimized out>, argv=<value optimized out>) at InitOutput.c:871
#5  0x000000000050daa8 in AddScreen (pfnInit=<value optimized out>, argc=5, argv=0x7fffffffe798) at dispatch.c:3919
#6  0x000000000041dccf in InitOutput (screenInfo=<value optimized out>, argc=5, argv=0x7fffffffe798) at InitOutput.c:970
#7  0x00000000004fc6ed in main (argc=5, argv=<value optimized out>, envp=<value optimized out>) at main.c:209
Comment 4 Christopher Yeleighton 2011-06-17 14:03:42 UTC
because depth[i].vids == NULL (when built with -O0)
this should be set in micmap.c:565
but it is not reached because miVisuals->count == 0 at micmap.c:563 (each time)
miVisuals is static at micmap.c:376, to be set at micmap.c:420
but then visuals == 0 (parameter), so count gets 0 too

now, miSetVisualTypes is called 
  * from miSetPixmapDepths (with visuals=0) 
  * from miInitVisuals, but only if (!miVisuals), which is not the case even if the server does not crash

However, it is successfully called from vfbScreenInit.

The code of vfbSceenInit (hw/vfb/InitOutput.c:832) explicitly supports some depths but not others:

<<<

    switch (pvfb->depth) {
    case 8:
	miSetVisualTypesAndMasks (8,
				  ((1 << StaticGray) |
				   (1 << GrayScale) |
				   (1 << StaticColor) |
				   (1 << PseudoColor) |
				   (1 << TrueColor) |
				   (1 << DirectColor)),
				  8, PseudoColor, 0, 0, 0);
	break;
    case 15:
	miSetVisualTypesAndMasks (15,
				  ((1 << TrueColor) |
				   (1 << DirectColor)),
				  8, TrueColor, 0x7c00, 0x03e0, 0x001f);
	break;
    case 16:
	miSetVisualTypesAndMasks (16,
				  ((1 << TrueColor) |
				   (1 << DirectColor)),
				  8, TrueColor, 0xf800, 0x07e0, 0x001f);
	break;
    case 24:
	miSetVisualTypesAndMasks (24,
				  ((1 << TrueColor) |
				   (1 << DirectColor)),
				  8, TrueColor, 0xff0000, 0x00ff00, 0x0000ff);
	break;
    case 30:
	miSetVisualTypesAndMasks (30,
				  ((1 << TrueColor) |
				   (1 << DirectColor)),
				  10, TrueColor, 0x3ff00000, 0x000ffc00, 0x000003ff);
	break;
    }

>>>

If the declared depth is not one of them, Xvfb crashes.
Comment 5 Christopher Yeleighton 2011-06-17 18:02:12 UTC
This particular code fragment is obviously wrong in not providing a default handler and it is not patched.