Wednesday, March 11, 2009

How to Generate Database Setup and Teardown MSBuild script with T4?

Problem

I been getting lots of emails on my previous article on using T4 template to generate Wix scripts by recursively looping through the source directory. I been using T4 template a lot lately in our continuous integration and I am beginning to replace lots of my XSL with T4 templates because C# is much easier to deal with then XSL. And I think this is purely personal preference. Also with Linq to XMl T4 template just got much more poweful.


References used to research and learn

Many feedbacks I received from the readers.
how to use msbuild and wix to msi


Assumption

1) Familiar with C#, XML, Linq
2) Familiar with MSBuild


About the project

Download: GenerateDatabaseSetupAndTeardownWithT4T.zip

I will cover how to generate MSBuild scripts to setup and tear down database. This is necessary if you truly practice continuous integration's one click deployment and automation.

1) How to quickly start using T4?
2) What is alternate way to debug?
3) Best practices we found that we want to share that worked well for us.

Here is very useful way to utilize T4 template in Continuous Integration



We first centralize everything in VARS.XML that contains all the configuration (dev, qa, uat, stg ect...). In the past I used XSL to spit out apps.config or web.config for different environment but with T4 Template it is much easier by templatizing variables and with XML to Linq it got that much easier to deal with.

Requirements

1) VS 2008
2) .Net 3.5 SP1


Step by step instruction

Create Console App Project



Add text file with .tt extension GenerateDatabaseSetupAndTeardown.tt

Click Ok to VS 2008 Warning



Here is the trick with T4 Template. Strength of T4 Template is that it is literally C# code with few exceptions. To get a quick start with T4 template in my opinion it is best to start with Console and or NUnit to create C# code. In this example Console app is used to spit out strings using Console.WriteLine in T4 Template it uses WriteLine. Now there is way to replace using C# variables but I am not going to cover that here. I taking the approach of typical programmer mindset and translate that into T4.

REMEBER TO REPLACE varsXml pointing to where vars.xml is.

using System;
using System.Text;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using System.Xml.Linq;
 
namespace GenerateDatabaseSetupAndTeardownWithT4T
{
  class Program
  {
    static void Main(string[] args)
    {
      XDocument doc;
      using (StreamReader s = File.OpenText(varsXml))
      {
        doc = XDocument.Load(s);
      }
 
      var dbs = from b in doc.Elements("Configurations")
                    .Elements("Global.Config")
                    .Elements("Databases")
                    .Elements("Database")
            where b.Attribute("ScriptThis").Value.ToLower().Equals("yes")
            select new
            {
              ScriptUser = b.Attribute("ScriptUser") == null ? null : b.Attribute("ScriptUser").Value,
              ScriptUserPassword = b.Attribute("ScriptUserPassword") == null ? null : b.Attribute("ScriptUserPassword").Value,
              Vars = b.Elements("Var"),
              Scripts = b.Elements("Scripts").Elements("Script")
            };
 
      List<string> sqlDataFileRoot = new List<string>();
      StringBuilder sb = new StringBuilder();
      foreach (var db in dbs)
      {
        string dataSource = string.Empty;
        string initialCatalog = string.Empty;
        string userID = db.ScriptUser;
        string password = db.ScriptUserPassword;
        string dropDB = string.Empty;
        string createDB = string.Empty;
        string currentSchema = string.Empty;
        string currentData = string.Empty;
 
        // extract from Var node
        foreach (XElement var in db.Vars)
        {
          if (var.Attribute("Name").Value.Equals("DataSource"))
          {
            dataSource = var.Attribute("Value").Value;
          }
          else if (var.Attribute("Name").Value.Equals("InitialCatalog"))
          {
            initialCatalog = var.Attribute("Value").Value;
          }
          else if (var.Attribute("Name").Value.Equals("SqlDataFileRoot"))
          {
            sqlDataFileRoot.Add(var.Attribute("Value").Value);
          }
        }
 
        // extract from Script node
        foreach (XElement script in db.Scripts)
        {
          if (script.Attribute("Name").Value.Equals("DropDB"))
          {
            dropDB = script.Attribute("Value").Value;
          }
          else if (script.Attribute("Name").Value.Equals("CreateDB"))
          {
            createDB = script.Attribute("Value").Value;
          }
          else if (script.Attribute("Name").Value.Equals("CurrentSchema"))
          {
            currentSchema = script.Attribute("Value").Value;
          }
          else if (script.Attribute("Name").Value.Equals("CurrentData"))
          {
            currentData = script.Attribute("Value").Value;
          }
        }
 
        // Generate MSBuild that tears down the database based no
        if (!string.IsNullOrEmpty(userID) && !string.IsNullOrEmpty(password))
        {
          sb.AppendFormat(@"<Sql.GetDatabaseVersion{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""user id={0};password={1};data source={2};initial catalog={3};""{4}", userID, password, dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60"" {0}", Environment.NewLine);
          sb.AppendFormat(@"      ContinueOnError=""true"">{0}", Environment.NewLine);
          sb.AppendFormat(@"      {0}", Environment.NewLine);
          sb.AppendFormat(@"      <Output TaskParameter=""DatabaseVersion"" PropertyName=""{0}-DatabaseVersion""/>{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    </Sql.GetDatabaseVersion>{0}", Environment.NewLine);
          sb.AppendFormat(@"    <Sql.DisconnectUsers{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""user id={0};password={1};data source={2};initial catalog={3};""{4}", userID, password, dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60"" {0}", Environment.NewLine);
          sb.AppendFormat(@"      Condition= "" '$({0}-DatabaseVersion)' != '0.0' and '$({0}-DatabaseVersion)' != '' "" />{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    <Sql.Execute {0}", Environment.NewLine);
          sb.AppendFormat(@"      Path=""{0}"" {1}", dropDB, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""user id={0};password={1};data source={2};initial catalog=master;"" />{3}", userID, password, dataSource, Environment.NewLine);
 
        }
        else
        {
          sb.AppendFormat(@"<Sql.GetDatabaseVersion{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""Integrated Security=true;data source={0};initial catalog={1};"" {2}", dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60""{0}", Environment.NewLine);
          sb.AppendFormat(@"      ContinueOnError=""true"">{0}", Environment.NewLine);
          sb.AppendFormat(@"      {0}", Environment.NewLine);
          sb.AppendFormat(@"      <Output TaskParameter=""DatabaseVersion"" PropertyName=""{0}-DatabaseVersion""/>{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    </Sql.GetDatabaseVersion>{0}", Environment.NewLine);
          sb.AppendFormat(@"    <Sql.DisconnectUsers{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""Integrated Security=true;data source={0};initial catalog={1};"" {2}", dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60"" {0}", Environment.NewLine);
          sb.AppendFormat(@"      Condition= "" '$({0}-DatabaseVersion)' != '0.0' and '$({0}-DatabaseVersion)' != '' "" />{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    <Sql.Execute {0}", Environment.NewLine);
          sb.AppendFormat(@"      Path=""{0}""{1}", dropDB, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""Integrated Security=true;data source={0};initial catalog=master;"" />{1}", dataSource, Environment.NewLine);
 
        }
      }
 
      foreach (string dir in sqlDataFileRoot)
      {
        sb.AppendFormat(@"<RemoveDir Directories=""{0}"" />{1}", dir, Environment.NewLine);
      }
 
      Console.WriteLine(sb.ToString());
 
    }
  }
}


Once tested bring it over to GenerateDatabaseSetupAndTeardown.tt

<#@ template language="C#v3.5" #>
<#@ assembly name="System.Core.dll" #>
<#@ assembly name="System.Xml.dll" #>
<#@ assembly name="System.Xml.Linq.dll" #>
<#@ import namespace="System.Xml" #>
<#@ import namespace="System.Xml.Linq" #>
<#@ import namespace="System" #>
<#@ import namespace="System.IO" #>
<#@ import namespace="System.Text" #>
<#@ import namespace="System.Diagnostics" #>
<#@ import namespace="System.Collections.Generic" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Collections" #>
<#@ import namespace="System.Security.Cryptography" #>
<#@ output extension=".xml" #>
<#  
// string varsXml = @"[--VRSXML--]";
string varsXml = @"E:\repos\nas\src\blogs\GenerateDatabaseSetupAndTeardownWithT4T\GenerateDatabaseSetupAndTeardownWithT4T\vars.xml";
string dataSource;
string initialCatalog;
string userID;
string password;
 
GenerateScript(varsXml);
 
#>
 
<#+ private void GenerateScript(string varsXml) { #>
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="CleanDatabases">
<#+
      XDocument doc;
      using(StreamReader s = File.OpenText(varsXml))
      {
        doc = XDocument.Load(s);
      }
 
      var dbs = from b in doc.Elements("Configurations")
                    .Elements("Global.Config")
                    .Elements("Databases")
                    .Elements("Database")
              where b.Attribute("ScriptThis").Value.ToLower().Equals("yes")
              select new
              {
                ScriptUser = b.Attribute("ScriptUser") == null ? null : b.Attribute("ScriptUser").Value,
                ScriptUserPassword = b.Attribute("ScriptUserPassword") == null ? null : b.Attribute("ScriptUserPassword").Value,
                Vars = b.Elements("Var"),
                Scripts = b.Elements("Scripts").Elements("Script")
              };
 
      List<string> sqlDataFileRoot = new List<string>();
      StringBuilder sb = new StringBuilder();
      foreach (var db in dbs)
      {
        string dataSource = string.Empty;
        string initialCatalog = string.Empty;
        string userID = db.ScriptUser;
        string password = db.ScriptUserPassword;
        string dropDB = string.Empty;
        string createDB = string.Empty;
        string currentSchema = string.Empty;
        string currentData = string.Empty;
 
        // extract from Var node
        foreach (XElement var in db.Vars)
        {
          if (var.Attribute("Name").Value.Equals("DataSource"))
          {
            dataSource = var.Attribute("Value").Value;
          }
          else if (var.Attribute("Name").Value.Equals("InitialCatalog"))
          {
            initialCatalog = var.Attribute("Value").Value;
          }
          else if (var.Attribute("Name").Value.Equals("SqlDataFileRoot"))
          {
            sqlDataFileRoot.Add(var.Attribute("Value").Value);
          }
        }
 
        // extract from Script node
        foreach (XElement script in db.Scripts)
        {
          if (script.Attribute("Name").Value.Equals("DropDB"))
          {
            dropDB = script.Attribute("Value").Value;
          }
          else if (script.Attribute("Name").Value.Equals("CreateDB"))
          {
            createDB = script.Attribute("Value").Value;
          }
          else if (script.Attribute("Name").Value.Equals("CurrentSchema"))
          {
            currentSchema = script.Attribute("Value").Value;
          }
          else if (script.Attribute("Name").Value.Equals("CurrentData"))
          {
            currentData = script.Attribute("Value").Value;
          }
        }
 
        // Generate MSBuild that tears down the database based no
        if (!string.IsNullOrEmpty(userID) && !string.IsNullOrEmpty(password))
        {
          sb.AppendFormat(@"<Sql.GetDatabaseVersion{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""user id={0};password={1};data source={2};initial catalog={3};""{4}", userID, password, dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60"" {0}", Environment.NewLine);
          sb.AppendFormat(@"      ContinueOnError=""true"">{0}", Environment.NewLine);
          sb.AppendFormat(@"      {0}", Environment.NewLine);
          sb.AppendFormat(@"      <Output TaskParameter=""DatabaseVersion"" PropertyName=""{0}-DatabaseVersion""/>{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    </Sql.GetDatabaseVersion>{0}", Environment.NewLine);
          sb.AppendFormat(@"    <Sql.DisconnectUsers{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""user id={0};password={1};data source={2};initial catalog={3};""{4}", userID, password, dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60"" {0}", Environment.NewLine);
          sb.AppendFormat(@"      Condition= "" '$({0}-DatabaseVersion)' != '0.0' and '$({0}-DatabaseVersion)' != '' "" />{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    <Sql.Execute {0}", Environment.NewLine);
          sb.AppendFormat(@"      Path=""{0}"" {1}", dropDB, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""user id={0};password={1};data source={2};initial catalog=master;"" />{3}", userID, password, dataSource, Environment.NewLine);
 
        }
        else
        {
          sb.AppendFormat(@"<Sql.GetDatabaseVersion{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""Integrated Security=true;data source={0};initial catalog={1};"" {2}", dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60""{0}", Environment.NewLine);
          sb.AppendFormat(@"      ContinueOnError=""true"">{0}", Environment.NewLine);
          sb.AppendFormat(@"      {0}", Environment.NewLine);
          sb.AppendFormat(@"      <Output TaskParameter=""DatabaseVersion"" PropertyName=""{0}-DatabaseVersion""/>{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    </Sql.GetDatabaseVersion>{0}", Environment.NewLine);
          sb.AppendFormat(@"    <Sql.DisconnectUsers{0}", Environment.NewLine);
          sb.AppendFormat(@"      DatabaseName=""{0}""{1}",initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""Integrated Security=true;data source={0};initial catalog={1};"" {2}",dataSource, initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"      CommandTimeout=""60"" {0}", Environment.NewLine);
          sb.AppendFormat(@"      Condition= "" '$({0}-DatabaseVersion)' != '0.0' and '$({0}-DatabaseVersion)' != '' "" />{1}", initialCatalog, Environment.NewLine);
          sb.AppendFormat(@"    <Sql.Execute {0}", Environment.NewLine);
          sb.AppendFormat(@"      Path=""{0}""{1}", dropDB, Environment.NewLine);
          sb.AppendFormat(@"      ConnectionString=""Integrated Security=true;data source={0};initial catalog=master;"" />{1}", dataSource, Environment.NewLine);
 
        }
      }
 
      foreach(string dir in sqlDataFileRoot)
      {
        sb.AppendFormat(@"<RemoveDir Directories=""{0}"" />{1}", dir, Environment.NewLine);
      }
 
      WriteLine(sb.ToString());
#>
  </Target>
</Project>
<#+ } #>


When you Save .TT file you will see that it spits out MSBuild script

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="CleanDatabases">
<Sql.GetDatabaseVersion
      DatabaseName="MyProject1"
      ConnectionString="user id=sa;password=password007;data source=localhost;initial catalog=MyProject1;"
      CommandTimeout="60" 
      ContinueOnError="true">
 
      <Output TaskParameter="DatabaseVersion" PropertyName="MyProject1-DatabaseVersion"/>
    </Sql.GetDatabaseVersion>
    <Sql.DisconnectUsers
      DatabaseName="MyProject1"
      ConnectionString="user id=sa;password=password007;data source=localhost;initial catalog=MyProject1;"
      CommandTimeout="60" 
      Condition= " '$(MyProject1-DatabaseVersion)' != '0.0' and '$(MyProject1-DatabaseVersion)' != '' " />
    <Sql.Execute 
      Path="$(SqlRoot)DropMyDB1.sql" 
      ConnectionString="user id=sa;password=password007;data source=localhost;initial catalog=master;" />
<Sql.GetDatabaseVersion
      DatabaseName="MyProject2"
      ConnectionString="user id=sa;password=password321;data source=localhost;initial catalog=MyProject2;"
      CommandTimeout="60" 
      ContinueOnError="true">
 
      <Output TaskParameter="DatabaseVersion" PropertyName="MyProject2-DatabaseVersion"/>
    </Sql.GetDatabaseVersion>
    <Sql.DisconnectUsers
      DatabaseName="MyProject2"
      ConnectionString="user id=sa;password=password321;data source=localhost;initial catalog=MyProject2;"
      CommandTimeout="60" 
      Condition= " '$(MyProject2-DatabaseVersion)' != '0.0' and '$(MyProject2-DatabaseVersion)' != '' " />
    <Sql.Execute 
      Path="$(SqlRoot)DropMyDB2.sql" 
      ConnectionString="user id=sa;password=password321;data source=localhost;initial catalog=master;" />
<RemoveDir Directories="c:\sqldata\MyDB1" />
<RemoveDir Directories="c:\sqldata\MyDB2" />
 
  </Target>
</Project>
 


Conclusion

Before Linq I did not consider using T4 template for anything else but for Wix script generation but now with Linq to XML it is much easier to read and deal with XML then XSL in my opinion. And I get much more control with C#. Of course in typical SCM some people are not comfortable with C# so this approach I shown here definitely is not for everyone.

Saturday, February 28, 2009

Silverlight 2 Tutorials

Recently, we launched new site dedicated to help developers learn about Microsoft Silverlight 2 technology. The reason we are not hosting it on this blog is because it is difficult to deploy Silverlight 2 applications to blogger.com site :(

So we are hosting it on our server that allows us to easily demo Silverlight 2 applications.

Please visit Silverlight.NewAgeSolution.net to learn more about how to develop Silverlight 2 applications.

Monday, November 17, 2008

How to Block Country and IP Addresses in ASP.NET using HTTPModule?

Problem

Recently, we noticed in our web traffic report some very suspicious activities so we decide to block certain ip addresses. Now there are many ways block ip addresses like at Windows server level, using IIS and at ASP.NET level.

First two choices works only if the person had access to the server and it becomes problem if one does not have access to the server and blocking IP address in ASP.NET becomes only choice.

References used to research and learn

Custom configuration sections
Creating ASP.NET HTTPModule
Restful API for finding out IP address's country

Assumption

1) The reader is familiar with ASP.NET
2) Wants to learn about HTTPModule, and Custom Configuration Sections

About the project

Download: IPBlocker.zip source

Here are the features of IPBlocker:

0) Rule is if the IP address is in denied list or ip address is not in granted list it will be blocked. Also, IP address will be checked against all the defined blocking scheme in the configuration until ip is blocked. If the IP is granted access and not in blocked list IP address will be granted.

1) IP Address can contain * as wild character.
i.e. <ip value="127.0.0.1" access="granted" valueType="IpAddress" />

2) Range of IP address can be specified.
i.e. <ip value="127.0.0.1;127.0.0.200" access="denied" valueType="IpRange" />

3) Can block country.
i.e. <ip value="IN;US" access="denied" valueType="CountryCode" />

4) Multiple IP address blocking scheme can be defined.
i.e. grant all ip from 127.0.0.1 through 127.0.0.255 EXCEPT 127.0.0.20 through 127.0.0.25.
<ip value="127.0.0.*" access="granted" valueType="IpAddress" />
<ip value="127.0.0.20;127.0.0.25" access="denied" valueType="IpRange" />

Here is sample configuration



<IpBlockerGroup>


  <IpBlockerConfiguration>


 


    <pages defaultScheme="blockIp">


      <page name="contactus.aspx" />


      <page name="home.aspx" scheme="blockIpRange" />


      <page name="aboutus.aspx" scheme="blockCountry" />


    </pages>


 


    <schemes>


      <!-- Used for Load Test -->


      <scheme name="blockIp">


        <ips>


          <!-- Use * as wild card -->


          <ip value="*.*.*.*" access="granted" valueType="IpAddress" />


          <!-- min value and max value seperated by ; (i.e. 127.1.2.3;233.3.3.2) -->


          <ip value="127.0.0.1;127.0.0.2" access="denied" valueType="IpRange" />


          <!-- ; seperated country code (i.e. IN;US) -->


          <ip value="IN;US" access="denied" valueType="CountryCode" />


        </ips>


      </scheme>


    </schemes>


 


  </IpBlockerConfiguration>


</IpBlockerGroup>




Requirements

1) VS2008
2) .Net 3.5

Step by step instruction

In web.config, add to configSections:



<sectionGroup name="IpBlockerGroup">


  <section


    name="IpBlockerConfiguration"


    type="IpBlocker.Configuration.IpBlockerConfigurationSection, IpBlocker"


    allowLocation="true"


    allowDefinition="Everywhere" />


</sectionGroup>




Add to configuration



<IpBlockerGroup>


  <IpBlockerConfiguration>


    <pages defaultScheme="blockCountry">


      <page name="contactus.aspx" />


    </pages>


    <schemes>


      <!-- Used for Load Test -->


      <scheme name="blockCountry">


        <ips>


          <!-- Use * as wild card -->


          <!--<ip value="*.*.*.*" access="granted" valueType="IpAddress" />-->


          <!-- min value and max value seperated by ; (i.e. 127.1.2.3;233.3.3.2) -->


          <!--<ip value="127.0.0.1;127.0.0.2" access="denied" valueType="IpRange" />-->


          <!-- ; seperated country code (i.e. XX;IN;US) -->


          <ip value="IR;RU;SA;IN" access="denied" valueType="CountryCode" />


        </ips>


      </scheme>


    </schemes>


  </IpBlockerConfiguration>


</IpBlockerGroup>




That it for running the module. Let's look at what is happening at more detailed level.



public class IpBlockerModule : IHttpModule


{


  public IpBlockerModule()


  {


  }


 


  public void Init(HttpApplication context)


  {


    context.BeginRequest += new EventHandler(Context_BeginRequest);


  }


 


  public void Dispose()


  {


  }


 


  private void Context_BeginRequest(object sender, EventArgs e)


  {


    IpBlockerConfigurationSection config = ConfigurationManager.GetSection("IpBlockerGroup/IpBlockerConfiguration") as IpBlockerConfigurationSection;


    HttpApplication application = sender as HttpApplication;


    string aspxPageName = System.IO.Path.GetFileName(application.Request.Url.AbsolutePath).ToUpper();


 


    PageElement page = config.Pages[aspxPageName];


    if (page != null && config.Schemes[config.Pages[aspxPageName].Scheme].CanBlockIp(application.Request.UserHostAddress))


    {


      // Blocking process


      application.Response.StatusCode = 404;


      application.Response.SuppressContent = true;


      application.Response.End();


    }


  }


 


}




IpBlockerModule listens to BeginRequest and checks to see if IP is granted to proceed for the specific page. Notice that in configuration Every page can have different IP blocking scheme. The reason why IP is blocked on specific page is because page like contact page that has email submission can be protected while other pages are not.

Conclusion

This module was developed out of need because even with captcha we had our contact page visited by some very starge ips performing very strange things. Also, it was good practice in creating HTTPModule and creating custom configuration.

Sunday, November 2, 2008

How to perform basic SEO for the web site?

Problem

We would like to educate everyone with some of technique that we use to increase our customers web sites rank and traffic.

First, performing SEO on the web site is not new or is it kept secret that only our company knows about it.

Second, it takes understanding of what SEO tools are available out there and how to properly use those SEO tools make differences.

References used to research and learn

what Google says about SEO
Very useful SEO tool

Assumption

1) Users are not familiar with SEO and wants to know more.

Explanation

Google Analytics
In order to understand the traffic behavior we use Google Analytics and we provide access to all our customers so they can view the traffics as well. Google Analytics provides very rich information about the web traffics such as what key words are used by the users to come to the site, where the users are accessing the web site, what browsers, how long the users are on the web site, what the users are viewing and more.

Webmaster Tool
In order to further optimize the web sites we have we use Google webmaster tool which allows to see external links backlinks, what kind of key words Google sees, when Google indexed the web sites and more...

Google AdSense
Google AdSense provides many useful tools like how much the key words cost, typical search volume of the key words trying to SEO, and do advertising based on Click Per Cost model and more...

SEO check list
Then we go through the SEO check list for all our customers.

Content matter!
Ultimately, content really matters with SEO no matter what. We constantly analyze Google Analytics and Google web master tool and modify our customers' web site contents. Also we provide suggestions to our customers to add new contents in certain ways that the key words can standout.

Blog helps
Blog is one of the easiest way to drive traffics to the web sites especially if the blogs are useful. Also having useful blog can help create very valuable backlinks to very important sites. For example, one of the blog that we wrote has backlinks to Microsoft and other very important developers in the community. Having such backlink helps with SEO.

Conclusion

Some of our customers were contacted by unknown company promising all kinds of SEO utopia like creating backlinks to gurantee #1 rank. Google clearly states that NO ONE CAN guarantee #1 Google rank with specific key words. Google rank is something that takes lots of hard work using many of the tools provided above and constantly analyzing data and tweaking pages and also contributing relevant contents to the community using things like blog.

We are always email or phone call away from answering our customers' questions of what all this SEO stuff means :)

Sunday, September 21, 2008

How to deploy Silverlight 2 Beta 2 from CruiseControl.Net to IIS using MSBuild?

Problem

Deploying Silverlight 2 Beta 2 using MSBuild with CruiseControl.Net in Windows 2003 64bit is bit tricky.

Assumption

1) Familiar with MSBuild
2) Familiar with IIS 6
3) Using .Net 3.5 SP1
4) VS 2008

Step by step instruction

1) First error from CruiseControl.net was

(GetFrameworkPaths target) ->
2008-09-20 22:02:06,368 [Moo:DEBUG] C:\Program Files\MSBuild\Microsoft\Silverlight\v2.0\Microsoft.Silverlight.Common.targets(78,9): error : The Silverlight 2 SDK is not installed.

To Fix this just install Microsoft® Silverlight™ 2 Software Development Kit Beta 2

2) Use <Platform>AnyCPU</Platform> to compile

3) In CruiseControl.Net use %systemroot%\Microsoft.NET\Framework\v3.5\msbuild.exe to build.

4) In 64 bit, xap file does not get copied to Web Application project's ClientBin. So make sure to copy compiled xap file from Silverlight project's bin as shown below.
Configuration can be either Debug or Release build.


<Target

    Name="Silverlight-Hack">

 

    <PropertyGroup>

      <XapFile>$(AppRoot)Moo.Silverlight\bin\$(Configuration)\Moo.Silverlight.xap</XapFile>

      <CopyXapTo>$(AppRoot)Moo.Web\ClientBin\</CopyXapTo>

    </PropertyGroup>

 

    <MakeDir Directories="$(CopyXapTo)" Condition=" !Exists('$(CopyXapTo)') " />

 

    <Copy SourceFiles="$(XapFile)" DestinationFolder="$(CopyXapTo)" />

  </Target>



5) Make sure to add Silverlight 2 mime types in IIS

.xaml application/xaml+xml
.xap application/x-silverlight-app

6) Make sure add ClientAccessPolicy.xml to WCF service project if the Silverlight will be accessing the service from different application domain.
For more information using ClientAccessPolicy.xml


<?xml version="1.0" encoding="utf-8"?>

<access-policy>

  <cross-domain-access>

    <policy>

      <allow-from http-request-headers="*">

        <domain uri="*"/>

      </allow-from>

      <grant-to>

        <resource path="/" include-subpaths="true"/>

      </grant-to>

    </policy>

  </cross-domain-access>

</access-policy>



Conclusion

Hopely, when Silverlight 2 is released some of the bugs mentioned above will be addressed.

Saturday, September 20, 2008

How to create load generator for simulating concurent hits?

Problem
We had to quickly generate some load on BizTalk and we needed some quick poorman's load generator to do it. Also this load generator was required to process requests in batch.

Here is the requirements:
1) Generate load simulating x number of concurrent connections to WCF net.tcp hosted by BizTalk.
2) There is limited resources(memory) on the load generating machines.
3) Must maintain x number of concurrent connections.
4) Must simulate scale out.

Explanation of above requirements:
1) In BizTalk if receive port is cofigured to use WCF net.tcp there is setting to control concurrent connection and from client in WCF netTcpBinding maxConnections is used to configure the setting as shown below (line 65):


   60 <netTcpBinding>

   61   <binding name="Default_NetTcpBinding" closeTimeout="00:05:00"

   62     openTimeout="00:05:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"

   63     transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions"

   64     hostNameComparisonMode="StrongWildcard" listenBacklog="10"

   65     maxBufferPoolSize="524288" maxBufferSize="65536" maxConnections="10"

   66     maxReceivedMessageSize="65536">

   67     <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"

   68       maxBytesPerRead="4096" maxNameTableCharCount="16384" />

   69     <reliableSession ordered="true" inactivityTimeout="00:10:00"

   70       enabled="false" />

   71     <security mode="None">

   72       <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />

   73       <message clientCredentialType="Windows" />

   74     </security>

   75   </binding>

   76 </netTcpBinding>



2) For this poorman's generator ThreadPool.QueueUserWorkItem was used. The more threads were queued, more the application would consume memory and there was need for some threadshhold. There were many ways to achieve what we wanted, using Semaphore, BackgroundWorker, and publish and subscriber.

3) Given set period of time and given set of batch of request, we had to constantly push x number of messages to BizTalk.

4) Simulate some kind of scale out approach. So we need to be able to spawn multiple threads of this load generator.

References used to research and learn

I like using the resource below for quick reference.
My favorite resource

Assumption

1) This article will not focus on performance of thread pooling. It is simply load generator.

About the project

Download: PoormansLoadGenerator.zip

1) TestService.cs was used to wrap LoadGenerator.cs implementation. It was used for spawning the worker thread to simulate scaling out.

2) This simple generator was able to generate enough load on BizTalk and SQL server to bring CPU usage up to 80% performing 600 messages per second load. This number really depends on many variables like what BizTalk orchestration is doing, what messages, what stored procs are we executing, ect. For us this was the result we wanted using this load generator.

Requirements

.Net 3.5
VS2008

Step by step instruction

Let's first look at Load Generator's entry point.


class Program

{

    static void Main(string[] args)

    {

        IList<TestService> servicesToRun = new List<TestService>();

        for (int i = 0; i < Config.MaxLoadGenerators; i++)

        {

            LoadGenerator loadGenerator = new LoadGenerator();

            servicesToRun.Add(new TestService(loadGenerator));

        }

 

        Console.WriteLine("Starting the service...");

        foreach (TestService service in servicesToRun)

        {

            service.Start();

        }

 

        // Wait until key is pressed....

        Console.ReadLine();

 

 

        Console.WriteLine("Exiting the service...");

        foreach (TestService service in servicesToRun)

        {

            service.Shutdown();

        }

    }

}



Notice that Config.MaxLoadGenerators is used to control how many of instances of LoadGenerator can be running simultaneously. This is to control scaling out requirements on the single machine.


public class LoadGenerator :IService

{

    private static bool _shutDown;

 

    #region IService Members

 

    public void RegisterService()

    {

        _shutDown = false;

        ThreadPool.SetMaxThreads(Config.MaxConcurrentThread, Config.MaxConcurrentThread);

        ThreadPool.SetMinThreads(Config.MaxConcurrentThread, Config.MaxConcurrentThread);

    }

 

    public void Start()

    {

        int j = 0;

        long memory = GC.GetTotalMemory(true);

        long maxMem = Config.MaxMemoryToUse + memory;

 

        while (!_shutDown)

        {

            System.Threading.Thread.Sleep(Config.LoopWaitTime);

            memory = GC.GetTotalMemory(true);

            if (memory < maxMem)

            {

                for (int i = 0; i < Config.BatchSize; i++)

                {

                    string x;

                    string z;

                    SetData(out x, out z, i, j);

                    ThreadPool.QueueUserWorkItem(delegate(object notUsed)

                                {

                                    DoSomething(x, z);

                                });

                }

                ++j;

            }

        }

    }

 

    public void Shutdown()

    {

        _shutDown = true;

    }

 

    private void SetData(out string x, out string z, int i, int j)

    {

        x = "for = " + i;

        z = "while = " + j;

    }

 

    private static void DoSomething(string forStr, string whileStr)

    {

        // Put your implementations here

        System.Threading.Thread.Sleep(Config.DoSomethingSleepTime);

        Console.WriteLine(string.Format("{0}, {1}", forStr, whileStr));

 

    }

 

    #endregion

}



ThreadPool.SetMaxThreads is used to control maximum concurrent thread running at one point. ThreadPool.SetMinThreads is used so that it is always waiting to provide thread.

We were running things in batch mode and constantly pulling records from the SQL server only if the memory was avaiable and queing up the records queried from SQL server. Also, in order to maintain constant concurrent connections feeding BizTalk with messages we need to queue up the request so that as soon as the thread finishes its job there is thread available to be used.


public sealed class Config

    {

        public static int LoopWaitTime;

        public static int MaxConcurrentThread;

        public static long MaxMemoryToUse;

        public static int DoSomethingSleepTime;

        public static int MaxLoadGenerators;

        public static int BatchSize;

 

        private Config()

        {

        }

 

        static Config()

        {

            LoopWaitTime = 10;

            MaxConcurrentThread = 20;

            MaxMemoryToUse = 100000000; // 1 GB

            DoSomethingSleepTime = 500;

            MaxLoadGenerators = 2;

            BatchSize = 100;

        }

    }



There is enough configuration here to tweak the behavior of the load generator and by tweaking these configuration we were able to generate some big load (600 messages per seconds) while keeping the load generator simple and stable.

Conclusion

I think there are enough of commercial load generator including VS2008 Tester version simulating load. For quick and dirty this load generator is more than enough in my opinion.