Saturday, September 20, 2008

How to create load generator for simulating concurent hits?

Problem
We had to quickly generate some load on BizTalk and we needed some quick poorman's load generator to do it. Also this load generator was required to process requests in batch.

Here is the requirements:
1) Generate load simulating x number of concurrent connections to WCF net.tcp hosted by BizTalk.
2) There is limited resources(memory) on the load generating machines.
3) Must maintain x number of concurrent connections.
4) Must simulate scale out.

Explanation of above requirements:
1) In BizTalk if receive port is cofigured to use WCF net.tcp there is setting to control concurrent connection and from client in WCF netTcpBinding maxConnections is used to configure the setting as shown below (line 65):


   60 <netTcpBinding>

   61   <binding name="Default_NetTcpBinding" closeTimeout="00:05:00"

   62     openTimeout="00:05:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"

   63     transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions"

   64     hostNameComparisonMode="StrongWildcard" listenBacklog="10"

   65     maxBufferPoolSize="524288" maxBufferSize="65536" maxConnections="10"

   66     maxReceivedMessageSize="65536">

   67     <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"

   68       maxBytesPerRead="4096" maxNameTableCharCount="16384" />

   69     <reliableSession ordered="true" inactivityTimeout="00:10:00"

   70       enabled="false" />

   71     <security mode="None">

   72       <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />

   73       <message clientCredentialType="Windows" />

   74     </security>

   75   </binding>

   76 </netTcpBinding>



2) For this poorman's generator ThreadPool.QueueUserWorkItem was used. The more threads were queued, more the application would consume memory and there was need for some threadshhold. There were many ways to achieve what we wanted, using Semaphore, BackgroundWorker, and publish and subscriber.

3) Given set period of time and given set of batch of request, we had to constantly push x number of messages to BizTalk.

4) Simulate some kind of scale out approach. So we need to be able to spawn multiple threads of this load generator.

References used to research and learn

I like using the resource below for quick reference.
My favorite resource

Assumption

1) This article will not focus on performance of thread pooling. It is simply load generator.

About the project

Download: PoormansLoadGenerator.zip

1) TestService.cs was used to wrap LoadGenerator.cs implementation. It was used for spawning the worker thread to simulate scaling out.

2) This simple generator was able to generate enough load on BizTalk and SQL server to bring CPU usage up to 80% performing 600 messages per second load. This number really depends on many variables like what BizTalk orchestration is doing, what messages, what stored procs are we executing, ect. For us this was the result we wanted using this load generator.

Requirements

.Net 3.5
VS2008

Step by step instruction

Let's first look at Load Generator's entry point.


class Program

{

    static void Main(string[] args)

    {

        IList<TestService> servicesToRun = new List<TestService>();

        for (int i = 0; i < Config.MaxLoadGenerators; i++)

        {

            LoadGenerator loadGenerator = new LoadGenerator();

            servicesToRun.Add(new TestService(loadGenerator));

        }

 

        Console.WriteLine("Starting the service...");

        foreach (TestService service in servicesToRun)

        {

            service.Start();

        }

 

        // Wait until key is pressed....

        Console.ReadLine();

 

 

        Console.WriteLine("Exiting the service...");

        foreach (TestService service in servicesToRun)

        {

            service.Shutdown();

        }

    }

}



Notice that Config.MaxLoadGenerators is used to control how many of instances of LoadGenerator can be running simultaneously. This is to control scaling out requirements on the single machine.


public class LoadGenerator :IService

{

    private static bool _shutDown;

 

    #region IService Members

 

    public void RegisterService()

    {

        _shutDown = false;

        ThreadPool.SetMaxThreads(Config.MaxConcurrentThread, Config.MaxConcurrentThread);

        ThreadPool.SetMinThreads(Config.MaxConcurrentThread, Config.MaxConcurrentThread);

    }

 

    public void Start()

    {

        int j = 0;

        long memory = GC.GetTotalMemory(true);

        long maxMem = Config.MaxMemoryToUse + memory;

 

        while (!_shutDown)

        {

            System.Threading.Thread.Sleep(Config.LoopWaitTime);

            memory = GC.GetTotalMemory(true);

            if (memory < maxMem)

            {

                for (int i = 0; i < Config.BatchSize; i++)

                {

                    string x;

                    string z;

                    SetData(out x, out z, i, j);

                    ThreadPool.QueueUserWorkItem(delegate(object notUsed)

                                {

                                    DoSomething(x, z);

                                });

                }

                ++j;

            }

        }

    }

 

    public void Shutdown()

    {

        _shutDown = true;

    }

 

    private void SetData(out string x, out string z, int i, int j)

    {

        x = "for = " + i;

        z = "while = " + j;

    }

 

    private static void DoSomething(string forStr, string whileStr)

    {

        // Put your implementations here

        System.Threading.Thread.Sleep(Config.DoSomethingSleepTime);

        Console.WriteLine(string.Format("{0}, {1}", forStr, whileStr));

 

    }

 

    #endregion

}



ThreadPool.SetMaxThreads is used to control maximum concurrent thread running at one point. ThreadPool.SetMinThreads is used so that it is always waiting to provide thread.

We were running things in batch mode and constantly pulling records from the SQL server only if the memory was avaiable and queing up the records queried from SQL server. Also, in order to maintain constant concurrent connections feeding BizTalk with messages we need to queue up the request so that as soon as the thread finishes its job there is thread available to be used.


public sealed class Config

    {

        public static int LoopWaitTime;

        public static int MaxConcurrentThread;

        public static long MaxMemoryToUse;

        public static int DoSomethingSleepTime;

        public static int MaxLoadGenerators;

        public static int BatchSize;

 

        private Config()

        {

        }

 

        static Config()

        {

            LoopWaitTime = 10;

            MaxConcurrentThread = 20;

            MaxMemoryToUse = 100000000; // 1 GB

            DoSomethingSleepTime = 500;

            MaxLoadGenerators = 2;

            BatchSize = 100;

        }

    }



There is enough configuration here to tweak the behavior of the load generator and by tweaking these configuration we were able to generate some big load (600 messages per seconds) while keeping the load generator simple and stable.

Conclusion

I think there are enough of commercial load generator including VS2008 Tester version simulating load. For quick and dirty this load generator is more than enough in my opinion.

1 comment:

Anonymous said...

We are also trying to evaluate these three, and we have a small team and a small budget.

It would be very interesting to know how thoughtworks was going for you, since it seems to be the cheapest, but it also seems to be the least polished of the three.

This post came up in a google search, im sure other people are reading it also, so let us know :)