Bug 336069 - Memory leak when allocating many small strings/arrays.
Summary: Memory leak when allocating many small strings/arrays.
Status: RESOLVED FIXED
Alias: None
Product: Mono: Runtime
Classification: Mono
Component: GC (show other bugs)
Version: unspecified
Hardware: x86 Linux
: P5 - None : Normal
Target Milestone: ---
Assignee: Paolo Molaro
QA Contact: Mono Bugs
URL:
Whiteboard:
Keywords: Code
Depends on:
Blocks:
 
Reported: 2007-10-23 16:34 UTC by A S
Modified: 2007-10-25 21:43 UTC (History)
2 users (show)

See Also:
Found By: Development
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description A S 2007-10-23 16:34:17 UTC
Description of Problem:
Memory leak when allocating many small strings/arrays.

Steps to reproduce the problem:
1. Compile and run the attached program
2. Wait until the program crashes.


Actual Results:
The program crashes and its memory consumption constantly climbs until the crash.

Expected Results:
The program  should not crash and the GC should gather the unused objects.

How often does this happen? 
Always

Additional Information:
The program doesn't crash under .NET CLR that comes with .NET framework 3. The memory consumption is constantly reduced by the GC and it doesn't climb up consistently like with mono.
Additionally, you can see that if you replace all CreateRandomString() with CreateField() (which doesn't actually create a new string object) the program runs fine and doesn't crash on Mono.

Here is the program:
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
using System.Threading;
using System.Collections;

namespace LargeArrays
{
    class Program
    {
        static void Main(string[] args)
        {
            BinaryFormatter formatter = new BinaryFormatter();
            CreateRandomStrings();
            while (true)
            {
                if (Console.KeyAvailable)
                {
                    ConsoleKeyInfo keyInfo = Console.ReadKey(true);
                    if (keyInfo.Key == ConsoleKey.Enter)
                        break;
                }

                List<Hashtable> surrogates = new List<Hashtable>();
                for (int i = 0; i < 200; ++i)
                {
                    Hashtable surrogate = new Hashtable();
                    CreateLargeSurrogate(surrogate);
                    surrogates.Add(surrogate);
                }

                byte[] bytes;
                using (MemoryStream memStream = new MemoryStream())
                {
                    byte[] innerBytes = null;
                    using (MemoryStream innerStream = new MemoryStream())
                    {
                        formatter.Serialize(innerStream, surrogates);
                        innerBytes = innerStream.ToArray();
                    }
                    memStream.Write(innerBytes, 0, innerBytes.Length);
                    bytes = memStream.ToArray();
                }
                System.Console.WriteLine(bytes.Length);

                Thread.Sleep(300);
            }
        }

        private static void CreateLargeSurrogate(Hashtable hashTable)
        {
            for (int i = 0; i < NumOfFields; ++i)
            {
                if (random.NextDouble() > 0.5)
                    hashTable[CreateField()] = random.NextDouble();
                else
                    hashTable[CreateField()] = CreateRandomString();
            }

            for (int i = 0; i < NumOfIntLists; ++i)
            {
                List<int> intList = new List<int>();
                for (int j = 0; j < 10; ++j)
                    intList.Add(random.Next());
                hashTable[CreateField()] = intList;
            }

            for (int i = 0; i < NumOfArrayLists; ++i)
            {
                List<string> strList = new List<string>();
                for (int j = 0; j < 10; ++j)
                    strList.Add(CreateRandomString());
                hashTable[CreateField()] = strList;
            }

            for (int i = 0; i < NumOfSurrogateLists; ++i)
            {
                List<Hashtable> subSurList = new List<Hashtable>();
                for (int j = 0; j < 10; ++j)
                    subSurList.Add(CreateSubSurrogate());
                hashTable[CreateField()] = subSurList;
            }
        }

        private static Hashtable CreateSubSurrogate()
        {
            Hashtable result = new Hashtable();
            for (int i = 0; i < NumOfSubSurrogateFields; ++i)
            {
                if (random.NextDouble() > 0.5)
                    result[CreateField()] = random.NextDouble();
                else
                    result[CreateField()] = CreateRandomString();
            }

            return result;
        }

        private static string CreateField()
        {
            int index = random.Next(0, randomStrings.Length);
            return randomStrings[index];
        }

        private static void CreateRandomStrings()
        {
            for (int j = 0; j < randomStrings.Length; ++j)
            {
                string randomString = CreateRandomString();
                randomStrings[j] = randomString;
            }
        }

        private static string CreateRandomString()
        {
            StringBuilder sb = new StringBuilder();
            int len = random.Next(FieldNameMinLength, FieldNameMaxLength);
            char[] chars = new char[len];
            for (int i = 0; i < chars.Length; ++i)
            {
                int minChar = Convert.ToInt32('A');
                int maxChar = Convert.ToInt32('z');
                chars[i] = Convert.ToChar(random.Next(minChar, maxChar));
            }

            sb.Append(chars);
            string randomString = sb.ToString();
            return randomString;
        }

        private static Random random = new Random();
        private static string[] randomStrings = new string[256 * 1024];

        private const int NumOfFields = 200;
        private const int NumOfIntLists = 5;
        private const int NumOfArrayLists = 5;
        private const int NumOfSurrogateLists = 5;
        private const int NumOfSubSurrogateFields = 10;
        private const int FieldNameMinLength = 10;
        private const int FieldNameMaxLength = 30;
    }
}
Comment 1 Robert Jordan 2007-10-23 18:00:38 UTC
It looks like a leak in the serialization infrastructure. By commenting
out the serialization code the test is running "forever" on my system
(SUSE 10.2, x86, Mono 1.2.5.1) consuming about 59M all the time.
Comment 2 A S 2007-10-23 20:00:03 UTC
But how do you explain the fact that when you use CreateField() instead of CreateRandomString() everywhere except in CreateRandomStrings() (that is, you con't generate any new strings during the program's execution) it doesn't leak and crash?

Maybe the combination of both large objects (the byte arrays from serialization) and small strings causes the problem?
Comment 3 Robert Jordan 2007-10-23 20:13:42 UTC
It's the high count of objects that seems to trigger the leak in the
serailization code.

Maybe it employs some static caches that linearly grow with
the amount of objects being serialized so far.
Comment 4 A S 2007-10-23 21:49:09 UTC
This might be the cause. Also, please look here:
http://www.mail-archive.com/mono-list@lists.ximian.com/msg22436.html

Somebody already encountered a similar problem.
Does anybody know when will it be fixed or if it has already been fixed in the latest release? (we use Mono 1.2.3.1 for our application).
Comment 5 Forgotten User vxPDddArjq 2007-10-25 21:43:19 UTC
Should be fixed in SVN.