Bug 333292

Btw. that 1.3GB beagled was from running over night.  It seems that at 1:11 AM
during the night something happened, because I now have 39179 tmp$HEX.tmp
files in /tmp/ which I noticed come from beagled (obviously it's copying
over files it wants to index to /tmp/ first).  These 39179 were all created
between 1:11 AM and 1:14 AM.  They never were removed, not even when I stoped
that beagled.  And when I killed it later they of course also were left :-/

From looking at some of those files it seems they are all mail bodies.

I was going to say something similar some weeks ago but never got round to it. I installed 10.3 by downloading the CD and then going on line. A few days later I found a message about "disc full". I found that /tmp was full of files called /tmp*.zip (eg tmp473989b6.zip). I rm'ed them (which took a time) and rebooted: reiser had to correct a couple of hundred links on that disc and a few in a second drive.

This continued on a daily basis, ie /tmp filled up at random times of the day with /tmp*.zip files and I had to reboot after removing them, for several weeks and I thought that I had a virus of some sort: I was considering backing up everything and reinstalling from a purchased version of 10.3.

I eventually noticed that speeds had dropped very considerably and realised that beagle was indexing the files. With 10.2 I had turned beagle off: the installation of 10.3 had turned it on without my noticing the fact.

The problem was I think caused by the fact that I have two mounted NFS directories in my home directory. It may also have been caused by the fact that I backup files on this machine from a remote machine using   Backup::Snapback http://search.cpan.org/~mikeh/Snapback2-0.913/
which uses hard links to files. (I did not want any of those files indexed which is why I had turned off beagle...)

With me the tmp files were zips of about 165kb containing a number of zips with names like page1.zip of about 4gb each containing a 4 gb file called 0.dll reduce to 4mb (which presumably means that it contains nothing but spaces) with a timestamp of 2000-03-028 (which is many years before this computer was built...).

Shouldn't beagle be set by default to ignore nfs/smb files?

(Having disabled beagle again my computer is once again usable, and no more tmp files have appeared...)

Roger
  

(In reply to comment #0 from Michael Matz)
> I see this with openSuSE 10.3, RC3, i.e. the final release:
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 27793 matz      22   7 1310m 539m 4772 S    0 53.4 142:59.08 beagled

I spotted this bug and am curious. Do you use Thunderbird backend and/or Evolution-Data-Server backend ?

I don't know.  This is the output of beagle-info --list-backends:

KMail                (/usr/lib/beagle/BeagleDaemonLib.dll)
Files                (/usr/lib/beagle/BeagleDaemonLib.dll)
GaimLog              (/usr/lib/beagle/BeagleDaemonLib.dll)
IndexingService      (/usr/lib/beagle/BeagleDaemonLib.dll)
Tomboy               (/usr/lib/beagle/BeagleDaemonLib.dll)
Labyrinth            (/usr/lib/beagle/BeagleDaemonLib.dll)
Blam                 (/usr/lib/beagle/BeagleDaemonLib.dll)
Liferea              (/usr/lib/beagle/BeagleDaemonLib.dll)
Akregator            (/usr/lib/beagle/BeagleDaemonLib.dll)
KonquerorHistory     (/usr/lib/beagle/BeagleDaemonLib.dll)
KonqBookmark         (/usr/lib/beagle/BeagleDaemonLib.dll)
KNotes               (/usr/lib/beagle/BeagleDaemonLib.dll)
KOrganizer           (/usr/lib/beagle/BeagleDaemonLib.dll)
KAddressBook         (/usr/lib/beagle/BeagleDaemonLib.dll)
Kopete               (/usr/lib/beagle/BeagleDaemonLib.dll)
Konversation         (/usr/lib/beagle/BeagleDaemonLib.dll)

which presumably means, I don't use the Thunderbird or Evolution backend.
I have switched off beagle since a long time, as obviously nobody of the
authors of it really cares to do something about this.

NFS home dir matz?

> which presumably means, I don't use the Thunderbird or Evolution backend.
Hmm...

Do you use mbox style emails in KMail ? All those tmp files must have come from somewhere.

> I have switched off beagle since a long time, as obviously nobody of the
> authors of it really cares to do something about this.
Well thats definitely not true, at least not 100% :) I am part of an author but
I didnt know about this problem until I read it on a forum post :(

You asked about any sort of debugging aid - yes you can use heap-shot to figure
out where are all the memory allocated. Well, in case if you are still
interested, that is.

And yes. Recently I encountered another NFS related problem. I am not 100% sure that indexing NFS homedirs work as expected.

JPR: yes.
D Bera: Actually I don't use kmail at all.  But I do use mbox, yes.  When I
last looked at this or a similar problem with beagle, those tmp files seemed
to contain individual mails from my mboxes in ~/Mail/ (and I have many of
those).  OTOH the number of tmp files is not the real issue, it's that they
are left behind.  And the bigger issue is that there's some memory leak.
IIRC I also straced the processes a bit at that time, and it was fiddling
with my mboxes while sucking memory, so a theory would be that this backend
has some bug.  I don't remember anymore if I tested this theory by disabling
the mail backend.

Hmm, now that I talk about this it might be that I even created some other
bug report with some more info regarding strangenesses in the mail backend.

Thanks for the hint about heap-shot, I might give it a shot :) somewhen next week.

I tried to reproduce the problem of indexing mbox files in the filesystem but failed! I tried both 0.2.18 and 0.3.2 but beagle simply (and rightly) refuses to index the mbox files which it recognizes as application/mbox. Could you run beagle-extract-content on one of the mbox files (preferably smaller one) and redirect the output to a file and check its first few lines of output what mimetype beagle is recognizing ? As far as I know, beagle will index single email files on the disk but I dont know what it does with mbox style collection of emails.

beagle-extract-content doesn't want to deal with mbox'es it seems:

(trara is a small mbox containing just two mails)

% beagle-extract-content ~/Mail/trara
Debug: Loaded 53 filters from /usr/lib/beagle/Filters/Filters.dll
Debug: No filter for file:///suse/matz/Mail/trara (/suse/matz/Mail/trara) [application/mbox]
No filter for application/mbox

But I know for certain that beagle _does_ index those files, as I still have
the indexes from a year ago, i.e. searching for text with kerry does find
mails from those very mboxes, and clicking on a result starts kmail like:

# kmail --view mbox:/suse/matz/Mail/in-binutils/From\ 
binutils-return-52787-matz=suse.de%40sourceware.org%20 Thu Oct%20 4 02:02:31\ 
2007

I've restarted the beagle daemon, and it starts eating memory immediately,
it runs since 20 minutes and needs already 315MB.  It seems when I last year
just killed beagled and the index helper it left over half-invalid indexes,
as I can see this in the current logfiles:

Caught exception trying to execute Beagle.Daemon.DaemonInformationExecutor.  Sending error response
System.IO.FileNotFoundException: Could not find file "/tmp/beagle-matz-d940e0a2-cc60-49e0-a322-9dcaf30bf675/Indexes/KMailIndex/PrimaryIndex/_13iy.fnm".
  at System.IO.FileStream..ctor (System.String name, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000]

This file (_13iy.fnm) is indeed not there, but referenced from the PrimaryIndex/segments file, so I assume the index is in an inconsistent state.
I don't yet want to remove and rebuild the index as I first want to see
where beagled eats the memory.

Now the question of course if, how beagled could index the mbox files,
when beagle-extract-content refuses to extract it.  I can only assume that
there another pre-processing pass (filter, extra program, whatever) which
splits the mbox into individual mails, and indexes those.  That would match
the behaviour I've seen in the past (something extracting the individual
mails to /tmp/ files, and something else (index-helper I think) indexing
_those_ tmp files).

beagled btw. also takes up much CPU time, it seems to wake up very often
without need (from strace):

[pid   777] gettimeofday({1201533056, 950187}, NULL) = 0
[pid   777] futex(0xb7865ff0, FUTEX_WAKE, 1) = 0
[pid   777] clock_gettime(CLOCK_REALTIME, {1201533056, 950557164}) = 0
[pid   777] futex(0xb786600c, FUTEX_WAIT, 16451, {0, 99442836}) = -1 ETIMEDOUT (Connection timed out)

(ad nauseam)

Hmm, so, how can I do that heap-shot thingy?  It seems it's required that
a program is started with extra mono options for this.  That's okay as I
seem to be able to reproduce the leaking behaviour.  But where can I
specify options for the beagled process?

Ignore my last question, it seems I can simply edit the /usr/bin/beagled
shell script to hardcode the heap-shot option which it seems to deal with
already anyway.

Explanations one by one :)

(In reply to comment #10 from Michael Matz)
> beagle-extract-content doesn't want to deal with mbox'es it seems:
..
> But I know for certain that beagle _does_ index those files, as I still have
> the indexes from a year ago, i.e. searching for text with kerry does find
> mails from those very mboxes, and clicking on a result starts kmail like:
> 
> # kmail --view mbox:/suse/matz/Mail/in-binutils/From\ 
> binutils-return-52787-matz=suse.de%40sourceware.org%20 Thu Oct%20 4 02:02:31\ 
> 2007

It was a bug in the KMail backend long ago that it thought any mbox email in ~/Mail was its email. Its fixed since then. But this information is good, now I will try to debug KMail with mbox. BTW, you dont have *.index files in ~/Mail do you ?

> I've restarted the beagle daemon, and it starts eating memory immediately,
> it runs since 20 minutes and needs already 315MB.  It seems when I last year
> just killed beagled and the index helper it left over half-invalid indexes,
> as I can see this in the current logfiles:
> 
> Caught exception trying to execute Beagle.Daemon.DaemonInformationExecutor. 
> Sending error response
> System.IO.FileNotFoundException: Could not find file
> "/tmp/beagle-matz-d940e0a2-cc60-49e0-a322-9dcaf30bf675/Indexes/KMailIndex/PrimaryIndex/_13iy.fnm".
>   at System.IO.FileStream..ctor (System.String name, FileMode mode, FileAccess
> access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions
> options) [0x00000]
> 
> This file (_13iy.fnm) is indeed not there, but referenced from the
> PrimaryIndex/segments file, so I assume the index is in an inconsistent state.
> I don't yet want to remove and rebuild the index as I first want to see
> where beagled eats the memory.

Can you stop beagled, and then restart, let it run for some time (with leak) and share the log files with me ? I want to add a safeguard against such behaviours; beagled already auto-corrects against some kind of corrupt indexes but not all cases are covered.

I have a hunch regarding the memory leak, probably some gmime stream is not closed/disposed. The last time I looked at the KMail backend was more than a year ago :)

> beagled btw. also takes up much CPU time, it seems to wake up very often
> without need (from strace):
> 
> [pid   777] gettimeofday({1201533056, 950187}, NULL) = 0
> [pid   777] futex(0xb7865ff0, FUTEX_WAKE, 1) = 0
> [pid   777] clock_gettime(CLOCK_REALTIME, {1201533056, 950557164}) = 0
> [pid   777] futex(0xb786600c, FUTEX_WAIT, 16451, {0, 99442836}) = -1 ETIMEDOUT
> (Connection timed out)
> 
> (ad nauseam)

Its a faulty Monitor.Wait in mono. They know about it. Monitor.Wait wakes up every 100ms.
 
> Hmm, so, how can I do that heap-shot thingy?  It seems it's required that
> a program is started with extra mono options for this.  That's okay as I
> seem to be able to reproduce the leaking behaviour.  But where can I
> specify options for the beagled process?

pass --heap-shot to beagled. The starting script will take care of the right mono options.

I don't have *index files, no.  (I use pine for mail :-) ).  When was the behaviour of the kmail backend fixed?  I'm asking because of files like this
in /tmp/beagle-matz-$SOMEID/Indexes/KMailIndex/ :

% cat offset--suse-matz-Mail-trara
50127

which indeed is the size of ~/Mail/trara (I'm assuming this is for easy
skipping already indexed contents), so it's doing at least _something_ to
those files.  Btw. I have nothing against the kmail backend looking at my
mbox files (making it a general mbox backend), it just should be non-leaky :)

I'll attach the Logfiles from my last run.  It ran for about half an hour.
It didn't stop when stopped from the GUI (the logfile tells me, that it did
receive the request for stopping), I had to kill the index-helper (normal
kill was enough), and also the beagled process (kill was not enough, I waited
for some time, until I had to kill -9 it).

Created attachment 192025 [details]
Some log files

This is the ~/.beagle/Log/ directory from the said run.  Ignore the .nfs*
files, they are old and from a (ohh, still running) different IndexHelper
process.

Ok. Anytime beagled/indexhelper hangs, send them a SIGQUIT, that will produce a stacktrace in the same terminal they were started. Could you attach them too ?

Possibly beagle-search is running when you started beagled. Could you start from a terminal (beagled --fg) ? Beagle-search is flooding with a lot of useless requests and not even starting the backends. I want to see what happens when normal startup happens. Note to self: remember to add a check for index corruption at any stage of indexing and querying!

Oh and you are running 0.2.18 ... that was has quite a few related problems. Namely, certain crashes in the indexhelper (as is happening now) were known to hang it.

I have the hanging situation, and sending SIGQUIT shows this interesting
"garbage" plus backtrace (this is still from a beagled --bg --debug, so from the Logfile):

Beagle DEBUG: All workers have finished.  Exiting main loop.

Well, so it should stop, but doesn't, the next lines of the Log being:

Beagle DEBUG: Joining thread 1 of 2: EHT 24651 [24562 BeagleDaemon] Beagle.Util.Scheduler:Worker

(here it hanged, and I sent it a QUIT, and then these lines showed up:)

2-digit year; 2-digit year; 2-digit year; 2-digit year; 2-digit year; 2-digit year; 2-digit year; 2-digit year; 2-digit year; 2-digit year; 2-digit year; Full thread dump:
"" tid=0x0xb7cef8f0 this=0x0x21e00:
  at (wrapper managed-to-native) System.Threading.Thread.Join_internal (int,intptr) <0x00004>
  at (wrapper managed-to-native) System.Threading.Thread.Join_internal (int,intptr) <0xffffffff>
  at System.Threading.Thread.Join () <0x00012>
  at Beagle.Util.ExceptionHandlingThread.JoinAllThreads () <0x00200>
  at Beagle.Daemon.BeagleDaemon.DoMain (string[]) <0x00b81>
  at Beagle.Daemon.BeagleDaemon.Main (string[]) <0x00014>

"EHT 24651 [24562 BeagleDaemon] Beagle.Util.Scheduler:Worker" tid=0x0xb5fc9b90 this=0x0x183840:
  at (wrapper managed-to-native) GMime.Parser.g_mime_parser_construct_message (intptr) <0x00004>
  at (wrapper managed-to-native) GMime.Parser.g_mime_parser_construct_message (intptr) <0xffffffff>
  at GMime.Parser.ConstructMessage () <0x00020>
  at Beagle.Daemon.KMailQueryable.KMailMboxIndexableGenerator.GetNextIndexable () <0x00044>
  at AddGeneratorTask.DoTaskReal () <0x00047>
  at Task.DoTask () <0x000ff>
  at Beagle.Util.Scheduler.Worker () <0x01b4f>
  at Beagle.Util.ExceptionHandlingThread.ThreadStarted () <0x002be>
  at (wrapper delegate-invoke) System.MulticastDelegate.invoke_void () <0xffffffff>
  at (wrapper runtime-invoke) System.IO.StreamWriter.runtime_invoke_void (object,intptr,intptr,intptr) <0xffffffff>
  at (wrapper runtime-invoke) Beagle.Daemon.BeagleDaemon.runtime_invoke_void_string[] (object,intptr,intptr,intptr) <0xffffffff>

So, one thread is just waiting for the other one, which called
KMailMboxIndexableGenerator.GetNextIndexable.  Sending further QUITs
shows that this task doesn't come out of a loop calling GetNextIndexable()
(or a loop entered from that function).

The stuff before "Full thread dump:" is interesting, it reads "2-digit year;"
a couple of times, obviously some debug message mentioning something which
probably the code isn't able to deal with.  Perhaps that causes a
non-advancement of the iterator and hence an endless loop.

Created attachment 192037 [details]
A log file from the last beagle run

This logfile is the one containing the above backtrace messages.  beagled was
called with "beagled --debug --indexing-delay 0 --bg"

Great. At about 19:32:36 this happened ...
20080128 19:32:36.7752 24562 Beagle DEBUG: Opening mbox /suse/matz/Mail/procmail.log

and then there was a pause for about 11secs. Probably it hanged in gmime. Do you remember if there was also a CPU problem ? GMime sometimes hangs on invalid files - and definitely procmail.log is not a file that GMime should process.

This points to another bug ... the KMail backend is not supposed to ~/Mail unless it is certain that it is a KMail folder. I thought I took care of all the cases - but clearly 0.2.18 (I dont remember if some new fix went into 0.3.0) thinks ~/Mail is a kmail folder. KMail backend can happily index mbox files from anybody, except if there are such files like procmail.log then it should not process them.

So we are looking at two bugs here, gmime freaking out on a non-mbox file and kmail backend thinking that ~/Mail is a kmail folder when it is not. The memory usage and (if happens) temporary files in ~/tmp are artifacts of this. The index corruption is hard to handle but it should not cause memory blowup or anything.

Could you
(1) Share/email me procmail.log ? or try to run parse it using GMime ?
(2) ls -al /suse/matz/Mail/ ?

After some hassle I now also have a heap-shot installation, and I now know
that the leak is not in Mono objects :-(  I've run beagled like above but
with heap-shot, (--heap-shot --indexing-delay 0 --bg), and it does leak,
and it generates snap shots of the heap.  Some excerpts, together with
messages when it automatically creates a heap-shot:

Memory usage: VmSize=28.8 MB, VmRSS=8.6 MB,  GC.GetTotalMemory=348160
Memory usage: VmSize=34.2 MB, VmRSS=13.1 MB,  GC.GetTotalMemory=999424
Suspicious memory size change detected.  Sending SIGPROF to ourself (0)

Then it idles on a bit, crawling the mails a bit, taking up 56MB in that
time:

Memory usage: VmSize=57.1 MB, VmRSS=24.2 MB,  GC.GetTotalMemory=3502080
Memory usage: VmSize=57.1 MB, VmRSS=24.4 MB,  GC.GetTotalMemory=3940352
Suspicious memory size change detected.  Sending SIGPROF to ourself (9)
Memory usage: VmSize=57.1 MB, VmRSS=24.7 MB,  GC.GetTotalMemory=4055040
....
Memory usage: VmSize=57.1 MB, VmRSS=24.7 MB,  GC.GetTotalMemory=4231168
Memory usage: VmSize=150.6 MB, VmRSS=100.6 MB,  GC.GetTotalMemory=4288512
Suspicious memory size change detected.  Sending SIGPROF to ourself (10)

Woah...

Memory usage: VmSize=230.8 MB, VmRSS=151.6 MB,  GC.GetTotalMemory=3461120
Suspicious memory size change detected.  Sending SIGPROF to ourself (11)

Jeez...

Memory usage: VmSize=252.4 MB, VmRSS=190.8 MB,  GC.GetTotalMemory=3362816
Suspicious memory size change detected.  Sending SIGPROF to ourself (12)
Memory usage: VmSize=278.4 MB, VmRSS=237.2 MB,  GC.GetTotalMemory=3354624
Suspicious memory size change detected.  Sending SIGPROF to ourself (13)

Mommy...

So, we learn that something does take up memory, and that heap-shot 9
is before that, and heap-shot 13 after.  Looking at both with heap-shot-gui
isn't revealing, though.  In all cases the memory traced by it is about
1.4MB overall, the biggest in strings of about 700 to 800 Kb:

% heap-shot outfile.beagled_13.omap | sort -n -k 2 | tail -n 5
650 46800 System.Uri
25 51502 byte[]
1215 58320 Beagle.Daemon.FileSystemQueryable.LuceneNameResolver/NameInfo
241 120148 System.Collections.Hashtable/Slot[]
10150 827678 string

vs.

heap-shot outfile.beagled_9.omap | sort -n -k 2 | tail -n 5
536 38592 System.Uri
25 51502 byte[]
1215 58320 Beagle.Daemon.FileSystemQueryable.LuceneNameResolver/NameInfo
241 120148 System.Collections.Hashtable/Slot[]
9581 748164 string

So the leak must be somewhere else, but is definitely there.  I have a theory
based on the debug messages around where the VmSize starts misbehaving:

Memory usage: VmSize=57.1 MB, VmRSS=24.7 MB,  GC.GetTotalMemory=4231168
Opening mbox /suse/matz/Mail/trara
trara: Finished indexing 0 messages
Opening mbox /suse/matz/Mail/procmail.log
Memory usage: VmSize=150.6 MB, VmRSS=100.6 MB,  GC.GetTotalMemory=4288512

So, it happily parsed some mboxes without any event, and then comes to
the procmail.log file, not an mbox file but looking like one.  Splitting
it into individual mails makes no sense and it's conceivable that doing this
nevertheless generates funny problems.  E.g. the delimiter between mails
("\n\nFrom ") doesn't occur in that file, but it does have "^From " lines
(in fact very many).  Depending on the algorithm to split mbox files it could
go wild when presented with a procmail log file, which looks like so:

From kde-commits-bounces-+kde.org-matz=kde.org@kde.org  Thu Apr 27 18:14:28 2006
 Subject: extragear/multimedia/k3b/src
  Folder: /suse/matz/Mail/in-kde-cvs                               3445
From evandro.menezes@amd.com  Thu Apr 27 18:15:42 2006
 Subject: RE: String Functions for x86-64
  Folder: /var/spool/mail/matz                                     3294
From kde-core-devel-bounces-+matz=suse.de@kde.org  Thu Apr 27 18:19:34 2006
 Subject: Re: kde4 dbus state ?
  Folder: /suse/matz/Mail/in-kde-core-devel                        3430
...
and so on.  This roughly looks like a traditional mail envelope with
two mail headers.  But it has no body, and no envelope separator.  Perhaps
it's treating the rest of the file as headers, or even walks into a qudratic
behaviour (starting from each "^From" line, regarding the whole rest of file
as headers).  Or something.

In any case, it's not in GC memory, but somewhere else, probably in the
mailbox parser.

As I said above, the trouble is in gmime parser. GC.GetTotalMemory represents the amount of memory in mono, note that even when RSS jumped this value remained pretty constant.

Could we ask gmime to not freak out when given a non-mbox file ? I dont know if they will agree :( Or it could be recent gmime takes care of this problem. Jeffrey Stedfast <fejj@ximian.com> would know better.

This was a mid-air collision, but I didn't want to rewrite everything I wrote
in comment #19.  My procmail.log is 101 MB, I don't want to attach it to
this (public) bug report.  How would I go about trying what GMime does with
it?  It seems there are no readily available command line clients doing
something with libgmime.

Btw. I unknowingly lied in comment #13.  I _do_ have .*.index files in ~/Mail.
They were hidden of course, and I didn't notice them existing.  Now that I saw
them again I darkly remember that I did once start kmail, long ago.
That must have generated those indexes.  Including an .index for procmail.log
(obviously kmail has the same difficulty of thinking that it's a mbox file).

That would probably explain why the KMail indexer backend things this is a
kmail ~/Mail/ .  Probably some safety measures with regards to "invalid"
mbox files would be in order.  Some parts of ~/Mail/:

-rw-------   1 matz suse     50127 2007-08-02 13:47 trara
-rw-------   1 matz suse      1381 2007-08-02 13:47 .trara.index
-rw-r--r--   1 matz suse        45 2008-01-28 15:50 .trara.index.ids
-rw-r--r--   1 matz suse       133 2007-08-02 13:47 .trara.index.sorted

So .index and .index.sorted are from last year, probably I started kmail for
whatever reason at that time.  And .index.ids is from today 15:50, which
roughly corresponds to the kmail started at that time when I clicked on
a link in kerry (see comment #10).  In addition to quite some more mboxes in
there I also have these directories:
drafts, inbox, outbox, templates, trash, each having subdirs cur,new and tmp.
All from 2002, I think this might be from experiments with netscape
communicator or the like.

Nice. Thats fixes one of the bugs :) About the backend considering Mail/ as a valid folder. Now the thing left to do is figure out where is gmime going crazy. I have a leave for a couple of hours but I will see if I can cut and paste the gmime parsing code to give you a helper program. gmime/ in svn might have a test/ or examples/ directory processing mbox files. You might also want to ask Jeff - probably he will be able to give a ready answer.

it'd be useful if someone could send me a copy of this procmail.log file because I'm not sure what's actually going on.

You can email it to fejj@novell.com and I will keep it private (or you can attach it here, doesn't matter to me)

Thanks.

This could also be bug 339481 which we are in the process of releasing an update for.  The gmime test packages are at:
http://download.opensuse.org/repositories/GNOME:/test-updates:/10.3/openSUSE_10.3/

nah, it's not the same bug.

I'm not sure how to fix this. since the first line looks like a mbox From_ line and there's no blank lines to mark an end-of-headers, it just keeps parsing each line after the first as if it were a header.

File formats are not always designed to protect against the worst case. In my experience, most formats allow adversarially created malformed files that still conform to the spec.

My suggestion would be to that gmime ignores this bug and claim this is a user error. I can see if I can ask beagle to not ask gmime to parse this file in the first place. I am not too hopeful about that since kmail (at least initially) thought the file was an mbox file.

One definite improvement could be the kmail backend actually parses the .index summary files that kmail creates and get the information from there about the number of message files in the mbox file etc. mbox is not a widely used format with kmail and so this was never implemented.

For some reason time and again, beagle gets caught in a pathological corner case :)

Jeff: Perhaps some kind of artificial limit in numbers of headers would be
acceptable to increase robustness when presented with invalid data.  E.g.
10000 headers surely is enough for a real Mail :-)  (my procmail.log contains
1739486 lines)

It's of course conceptually ugly, but the same method as e.g. also image
readers use to lessen impact by malformed files, i.e. limiting the acceptable
resolution.

Michael: perhaps...


I started working on a patch the other night and actually got it to work (I think), by being more strict in what I accept as a header.

That fix actually makes GMime handle some broken messages better (e.g. mails that don't put a blank line before dumping base64 content or some such).

The second part of my fix makes the header parser abort if it finds a "From "-line.

This, too, fixes one case in my broken mbox file where a message used a non-end-boundary to mark the end of its multipart and another message got butted up against it, like so:


Content-Type: multipart/mixed; boundary="xyz"

--xyz
Content-Type: ...

...
--xyz
Content-Type: ...

...
--xyz
From foo@bar.com <date>
<message headers>

that last "--xyz" should have been "--xyz--", but since it's not, GMime happily chugs along into the "From "-line and following message headers as if it were headers of the next mime part.

Now, normally, mbox files put a blank line between between messages - and if that were done in the above example, GMime would have parsed it properly - but since no blank line, it doesn't. So the "real" bug in the above case is that the mailer that wrote this mbox didn't properly append a blank line.


Anyways, I'll need at least this weekend to ponder my fix some more and maybe clean it up a bit more to be confident that it won't break anything else - if I'm satisfied, I'll commit it and append the final patch here.


I agree with D Bera, however, that ideally beagle shouldn't point GMime at a file and ask it to be parsed as an mbox w/o knowing for sure that it is an mbox file... then again, I'm not sure of any concrete way of knowing without trying to parse it ;-)


Reassigning to myself so that I can more easily find this bug again...

Created attachment 192906 [details]
bnc-333292.patch

This is a diff between gmime trunk and gmime-2.2.15

I just noticed SuSE 10.3 ships 2.2.10, but I don't think there have been any other gmime-parser.c changes since 2.2.10, so this patch should apply cleanly.

The result of parsing a procmail.log file should be basically 1 empty text/plain message per log entry after this patch.

Also fixes some other bugs wrt broken MIME messages in broken mbox files.

reassigning to default owner now that I've attached a patch

Michael, did you manage to test Jeff's patch ? Any more leaks ?

Changing to component GNOME. Sorry for the spam.

For now I'll close this as NORESPONSE.  Matz, please feel free to reopen this if you get more info (if you can send me that monstruous procmail.log, I can try to reproduce this...).

A C or C++ test program that uses libgmime to just parse a file would
be nice to have.  An example procmail.log is really easy to produce:

% echo > bla1.log <<EOF
From kde-commits-bounces-+kde.org-matz=kde.org@kde.org  Thu Apr 27 18:14:28 2006
 Subject: extragear/multimedia/k3b/src
  Folder: /suse/matz/Mail/in-kde-cvs                                      3445
EOF
% for i in `seq 1 6`; do
    cat bla1.log bla1.log bla1.log bla1.log > bla2.log
    cat bla2.log bla2.log bla2.log bla2.log > bla1.log
  done

That should give you a procmail logfile containing roughlyu 780000 lines,
a third of my own one, but probably enough to see any bad behaviour in
gmime.

I thought Jeff already incorporated his patch from #29 into gmime. So this problem should be fixed with any of the recent gmime releases.