In my previous post about exception logging, I show how to log several different parameters related to the exception in the database. Request.Browser.Crawler is one of them and its used to track browser crawlers. It warrants its own separate entry since it requires some extra bit of setup in the web.config to get it to work correctly.
You’ll have to add the following code in the section of your web.config file:
Now what does it all mean? Well, IIS uses that information in the section of your config file to detect whether the client browser is a crawler or not. If you look at it closely, its basically a regular expression filter. I presume you could add more filters in a similar format to detect other kinds of crawlers.
Update: For the most accurate and updated version of browserCaps and other useful browser testing/detection resources you can go to one of these sites: