Bug 327794

Summary: sometimes beagle logs way too much
Product: [openSUSE] openSUSE 10.2 Reporter: Martin Vidner <mvidner>
Component: OtherAssignee: E-mail List <gnome-bugs>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P2 - High CC: dbera.web, rainer
Version: Final   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 341832    
Attachments: IndexHelper-topbottom

Description Martin Vidner 2007-09-24 13:48:35 UTC
I found the bug on 10.2 but with the new Beagle 0.2.18 from http://software.opensuse.org/download/Beagle/openSUSE_10.2 so it is probably present in 10.3 as well.

mvidner@valkyrie:$ cd ~/.beagle/Log; LANG=C du -sc * | sort -rn | head
882396  total
418140  2007-09-20-09-53-21-IndexHelper
318440  2007-09-20-09-53-21-IndexHelperExceptions
40228   2007-09-21-09-16-41-IndexHelper
25428   2007-09-18-09-59-45-IndexHelper
23352   2007-09-24-09-43-46-IndexHelper
11172   2007-09-20-09-51-51-Beagle
10868   2007-09-20-09-51-56-Beagle
10440   2007-09-17-12-38-14-IndexHelper
8608    2007-09-20-09-51-51-BeagleExceptions

As you see sometimes the logs get way out of hand. The problem is probably magnified by the size of my mail archive (1.4GB).

Most of the messages in the logs are "IndexH DEBUG". I looked around for an option to turn the debugging off but could not find it.
BTW how does Beagle rotate the logs?

The largest files are at login.suse.cz:~mvidner/Export/big-{IndexHelper,IndexHelperExceptions}

Anyway, thanks for Beagle!
Comment 1 Joe Shaw 2007-09-24 14:51:33 UTC
The issue actually probably isn't present in 10.3 -- the version in OBS intentionally runs at log level "debug" whereas the versions we ship in the distro are only at log level "warn".  This is generally because the people who use the OBS version are either (a) interested parties who want to use the latest stuff or (b) people who upgraded because they had problems with older versions.  In the case of (b), the higher log level helps us identify the bug if it's still present or see if it's been fixed.

Having said that, an IndexHelper file that big, and a corresponding IndexHelperExceptions file may point to a bug.  Can you attach the first 1000 and last 1000 lines of 2007-09-20-09-53-21-IndexHelper?

Beagle deletes logs older than 7 days when it's restarted.  Size isn't a factor in log rotation.
Comment 2 Martin Vidner 2007-09-24 15:05:09 UTC
Created attachment 174375 [details]
IndexHelper-topbottom

I see. Yes there are exceptions in the log. It may be related that Beagle leaves an awful lot of tmp*.tmp files in /tmp. 14000 of them, 10000 of those from the past week.
Comment 3 Joe Shaw 2007-09-24 17:53:41 UTC
Yeah, and what's also odd is that it says it can't find a lot of those tmp files.  Did you perhaps run out of disk space on your /tmp partition?

(You can and should delete those tmp files if Beagle isn't running; I'm not sure why it didn't clean them up itself.)
Comment 5 Rainer Duffner 2008-04-23 09:08:48 UTC
Hi,

I also have this problem, on OpenSuSE 10.3 AMD64 (C2D 1.8 with 4GB RAM).
I just had to delete two 17 GB files out of the log-directory, filling up the rest of my /home almost to the last byte.

It logs a massive amount of exceptions, most seem to be related to Thunderbird.



20080423 10:53:27.4755 27665 Beagle  WARN EX: Caught exception in DoTaskReal
20080423 10:53:27.4755 27665 Beagle  WARN EX:         Tag: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/ip-exchange.msf
20080423 10:53:27.4755 27665 Beagle  WARN EX:     Creator: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/ip-exchange.msf
20080423 10:53:27.4755 27665 Beagle  WARN EX: Description: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/ip-exchange.msf
20080423 10:53:27.4755 27665 Beagle  WARN EX:    Priority: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/ip-exchange.msf ()
20080423 10:53:27.4755 27665 Beagle  WARN EX: System.UriFormatException: Invalid URI: The hostname could not be parsed.
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at System.Uri.Parse (System.String uriString) [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at System.Uri.ParseUri () [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at System.Uri..ctor (System.String uriString, Boolean dontEscape) [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at System.Uri..ctor (System.String uriString) [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at Beagle.Util.Thunderbird.NewUri (Beagle.Util.Account account, System.String mailbox, System.String id) [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at Beagle.Util.Thunderbird+Mail..ctor (Beagle.Util.Account account, System.Collections.Hashtable data, System.String workfile) [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at Beagle.Util.Thunderbird+Database+DatabaseEnumerator.get_Current () [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at Beagle.Daemon.ThunderbirdQueryable.MailIndexableGenerator.HasNextIndexable () [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at Beagle.Daemon.LuceneQueryable+AddGeneratorTask.DoTaskReal () [0x00000] 
20080423 10:53:27.4755 27665 Beagle  WARN EX:   at Beagle.Util.Scheduler+Task.DoTask () [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX: Caught exception in DoTaskReal
20080423 10:53:27.9914 27665 Beagle  WARN EX:         Tag: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/Drafts.msf
20080423 10:53:27.9914 27665 Beagle  WARN EX:     Creator: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/Drafts.msf
20080423 10:53:27.9914 27665 Beagle  WARN EX: Description: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/Drafts.msf
20080423 10:53:27.9914 27665 Beagle  WARN EX:    Priority: /home/rduffner/.thunderbird/kav1ydqw.default/ImapMail/imap.ultra-secure.de/INBOX.sbd/Drafts.msf ()
20080423 10:53:27.9914 27665 Beagle  WARN EX: System.UriFormatException: Invalid URI: The hostname could not be parsed.
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at System.Uri.Parse (System.String uriString) [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at System.Uri.ParseUri () [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at System.Uri..ctor (System.String uriString, Boolean dontEscape) [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at System.Uri..ctor (System.String uriString) [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at Beagle.Util.Thunderbird.NewUri (Beagle.Util.Account account, System.String mailbox, System.String id) [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at Beagle.Util.Thunderbird+Mail..ctor (Beagle.Util.Account account, System.Collections.Hashtable data, System.String workfile) [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at Beagle.Util.Thunderbird+Database+DatabaseEnumerator.get_Current () [0x00000] 
20080423 10:53:27.9914 27665 Beagle  WARN EX:   at Beagle.Daemon.ThunderbirdQueryable.MailIndexableGenerator.HasNextIndexable () [0x00000] 

Is there a way to fix it, apart from not running beagle at all (or having him not index Thunderbird-stuff)?
Comment 6 D Bera 2008-04-27 13:03:09 UTC
(In reply to comment #5 from Rainer Duffner)
> Hi,
> 
> I also have this problem, on OpenSuSE 10.3 AMD64 (C2D 1.8 with 4GB RAM).
> I just had to delete two 17 GB files out of the log-directory, filling up the
> rest of my /home almost to the last byte.
...
> Is there a way to fix it, apart from not running beagle at all (or having him
> not index Thunderbird-stuff)?

Upgrade to a more recent version (you are probably running beagle 0.2.18). The thunderbird backend was disabled upstream somewhere between 0.2.16 and 0.2.18 and replaced by a completely new one in 0.3.x; I am surprised you didnt find any other problems with the beagle version you have.
Comment 7 JP Rosevear 2008-09-12 00:44:28 UTC
The thunderbird backend is renabled and re-written.  Confirming we do inded default to log level warn on products.  I don't think there is much left to chase here now.