Bug 326808

Summary: beagle-index build create incomplete index (but doesn't fail)
Product: [openSUSE] openSUSE 10.3 Reporter: Stephan Binner <stbinner>
Component: GNOMEAssignee: Joe Shaw <joeshaw>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P5 - None CC: coolo
Version: RC 1Flags: coolo: SHIP_STOPPER-
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: Development Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Stephan Binner 2007-09-20 13:20:27 UTC
The build of latest the beagle-index version fails internally because Beagle crashes (see below). Second problem, the beagle-index build doesn't fail nevertheless and creates an 2.5MB big rpm (compared to a normal size of about 18MB nowadays):

Debug: +file:///usr/share/doc/packages/opensc/ReplacingCertificates.html
Stacktrace:

  at (wrapper managed-to-native) Beagle.Util.XdgMime.xdg_mime_get_mime_type_for_file (string,intptr) <0x00004>
  at (wrapper managed-to-native) Beagle.Util.XdgMime.xdg_mime_get_mime_type_for_file (string,intptr) <0xffffffff>
  at Beagle.Util.XdgMime.GetMimeType (string) <0x00033>
  at Beagle.Daemon.FilterFactory.FilterIndexable (Beagle.Indexable,Beagle.Daemon.TextCache,Beagle.Daemon.Filter&) <0x00144>
  at Beagle.Daemon.LuceneIndexingDriver.Flush_Unlocked (Beagle.Daemon.IndexerRequest) <0x0119e>
  at Beagle.Daemon.LuceneIndexingDriver.Flush (Beagle.Daemon.IndexerRequest) <0x00027>
  at Beagle.Daemon.BuildIndex.FlushIndexer (Beagle.Daemon.IIndexer,Beagle.Daemon.IndexerRequest) <0x00027>
  at Beagle.Daemon.BuildIndex.AddToRequest (Beagle.Daemon.IndexerRequest,Beagle.Indexable) <0x00177>
  at Beagle.Daemon.BuildIndex.DoIndexing () <0x0055e>
  at Beagle.Daemon.BuildIndex.IndexWorker () <0x00031>
  at Beagle.Daemon.BuildIndex.DoMain (string[]) <0x01367>
  at Beagle.Daemon.BuildIndex.Main (string[]) <0x00014>
  at (wrapper runtime-invoke) Beagle.Daemon.BuildIndex.runtime_invoke_void_string[] (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

        beagle-build-index [0x8170bea]
        beagle-build-index [0x813b3b2]
        [0xffffe600]
        /usr/lib/beagle/libbeagleglue.so(_xdg_mime_cache_unalias_mime_type+0x9c) [0xb703f30c]
        /usr/lib/beagle/libbeagleglue.so(_xdg_mime_unalias_mime_type+0x2e) [0xb703bf7e]
        /usr/lib/beagle/libbeagleglue.so(_xdg_mime_mime_type_equal+0x2f) [0xb703bfdf]
        /usr/lib/beagle/libbeagleglue.so(xdg_mime_mime_type_equal+0x29) [0xb703c519]
        /usr/lib/beagle/libbeagleglue.so [0xb703ff76]
        /usr/lib/beagle/libbeagleglue.so(_xdg_mime_cache_get_mime_type_for_file+0x13f) [0xb704016f]
        /usr/lib/beagle/libbeagleglue.so(xdg_mime_get_mime_type_for_file+0x61) [0xb703c6a1]
        [0xb6547bf3]
Comment 2 Joe Shaw 2007-09-20 15:07:51 UTC
Taking a look at it.  Our xdgmime is up-to-date with the one in GTK+, and I wasn't able to replicate the problem locally.  I'm trying to do an i386 build of beagle-index now.
Comment 3 Joe Shaw 2007-09-20 20:21:52 UTC
I think the segfault is caused by running out of memory:

99       32259 61.8  2.3 2053716 24124 pts/54  TNl+ 22:06   7:37 beagle-build-index /usr/lib/beagle/BuildIndex.exe --target /var/tmp/beagle-index-10.3_20070203-build/usr/share/beagle/mergables/documentation --

(ie, 2 gigs of RAM have been allocated.)  Attempting to malloc inside gdb also triggers a segfault.

I've still been unable to trigger it outside of an RPM build, though, which is odd.
Comment 4 Joe Shaw 2007-09-20 20:39:02 UTC
Well, malloc() even under reasonable memory crashes gdb.  Of course, I'm running this on macintyre which is SUSE Linux 10.0.  So, that's not that surprising.

Anyone have a 10.2 box I can ssh into and try?
Comment 5 Joe Shaw 2007-09-20 21:22:34 UTC
Ok, the memory thing was a red herring.  There is a problem with xdgmime's cache returning a NULL mime type, but I have no idea why right now (and a -O2 beagle isn't helping).
Comment 6 Joe Shaw 2007-09-21 18:57:49 UTC
Looks like repeated directories in the XDG_DATA_DIRS env var is causing this:

(gdb) print xdg_data_dirs
$50 = 0x7fff0107ee55 "/usr/share:/etc/opt/kde3/share:/opt/kde3/share:/usr/share:/etc/opt/kde3/share:/opt/kde3/share"

xdgmime is then thinking that the directory has been updated and is flushing its in-memory caches.  It looks like there's a race and xdgmime is poking at freed data.
Comment 7 Joe Shaw 2007-09-21 19:31:48 UTC
There are two bugs here, both in xdgmime.  I've filed them upstream:

https://bugs.freedesktop.org/show_bug.cgi?id=12512
https://bugs.freedesktop.org/show_bug.cgi?id=12513

The first contains a patch which will prevent the crash, but the second one illustrates a larger problem: xdgmime unnecessarily reloads its caches if a directory is repeated in XDG_DATA_DIRS, and likely creates performance problems.

xdgmime is cut-and-pasted into a variety of software like gtk+, nautilus, and probably some KDE stuff in addition to Beagle.  All of these would need to be patched.  We're better off finding out why XDG_DATA_DIRS has duplicates in 10.3 (and not in 10.2, which is why this crash is new).
Comment 8 JP Rosevear 2007-09-21 20:58:06 UTC
Could be the /etc/profile.d/xdg-environment.sh change for bug 300678.  FWIW, XDG_DATA_DIRS is not duped from me on a GNOME login.
Comment 9 JP Rosevear 2007-09-21 20:58:51 UTC
Actually, it is - /usr/local/share is in there twice.
Comment 10 Joe Shaw 2007-09-21 21:01:08 UTC
For me, XDG_DATA_DIRS repeats directories inside the RPM %build section, but not in a GNOME shell in a Beta 3 install.  So maybe this is only really a problem when building the beagle-index package.

I've submitted updated beagle packages (with the xdgmime patch) and beagle-index (to abort the build if it encounters an error)
Comment 11 Gary Ekker 2008-03-26 18:07:57 UTC
Changing to component GNOME. Sorry for the spam.