IIS / Tools / Log Files

Google Analytics - Track page views and analyze the flow of visitors

Chris Crowe — Thu, 12 Apr 2007 16:12:00 GMT

Google Analytics provides powerful tracking for anyone with a web presence, whether it be a small hobby website or a giant online enterprise. It's one of the most powerful web analytics solutions on the market - and it's free for anyone to use!

Google Analytics provides in-depth reports for everyone involved in the running of a website, from the developers and designers to the marketing and management teams. Find out where people leave your site, and what content catches their attention. Compare marketing campaigns and see your return on investment for all of your AdWords spend. The ways that you can use Google Analytics are endless.

This free version is limited to 5 million pageviews a month - however, users with an active Google AdWords account are given unlimited pageview tracking. In addition, Google Analytics is completely integrated into the AdWords front-end and with your AdWords campaign, making it easy to track your AdWords ROI.

http://www.google.com/analytics/

You can awatch a simple video here about how it works:
http://www.google.com/analytics/media/report_tour/feature_tour.html

Microsoft releases a new download center for IIS (everything in one place)

Chris Crowe — Wed, 31 Jan 2007 23:36:00 GMT

DownloadCENTER for IIS.net has been released!

The DownloadCENTER at IIS.net, is a community hotspot for discovering, sharing, reviewing and promoting IIS-related solutions in a single place. Dozens of existing downloads, for all versions of IIS – both from Microsoft and the community – are already available in DownloadCENTER today.

This new feature of IIS.net is particularly relevant with the release of IIS7 in Windows Vista. The latest release of Microsoft’s Web server has a completely modular architecture which features over forty pluggable components that can be easily added, removed or even replaced with custom implementations.

This powerful extensibility support is available to both .NET and C/C++ developers. In the future, DownloadCENTER is expected to house a large number of IIS7 extensions submitted by not only the IIS team but the developers and partner ISVs of the IIS community as well.

To learn more about the DownloadCenter, read IIS Product Unit Manager, Bill Staples’ blog post about it or check it out yourself today!

IIS 6 - Web Site Instance IDs - seemingly random but they are not!

Chris Crowe — Wed, 03 May 2006 11:27:00 GMT

On IIS 6.0 you will find that when you create web sites that their log files are created with seemingly random numbers. In previous versions of IIS the Instance IDs as they are called were sequentially numbered.

The first "Default Web Site" has a log file directory of W3SVC1 but any new sites you create may end up as follows (or something similar)

W3SVC1240841244
W3SVC1289352529
W3SVC147076792
W3SVC1527186048
W3SVC1566259604
W3SVC1710104836

I have written a script that you can use to display all the web sites and their associated Instance Ids - the output similar to the following.

Web Sites                    Description
===============================================================
W3SVC/1                   Default Web Site
W3SVC/1036328378 WebSite1
W3SVC/1816184000 WebSite2
W3SVC/1867813904 WebSite3
W3SVC/568530179   WebSite4
W3SVC/719499532   WebSite5
W3SVC/669732006   WebSite6

See http://blog.crowe.co.nz/archive/2005/12/08/346.aspx for the script.

Something I only learned recently was that the Instance Ids can be made sequential again by editing the registry and they are not random at all.

Changing the behaviour to the same as previous IIS versions

Remember : Editing the registry is a risk you take on your own shoulders.

To do this:

Click Start, click Run. In the Open box, type regedit, and then click OK.
In Registry Editor, locate the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\InetMgr\Parameters
Click the Parameters key. On the Edit menu, click New Dword Value.
Type IncrementalSiteIDCreation for the name.
Double-click the name of the new value.
Change Value Data to 1, and then click OK.
To enable the change, restart the IIS services.

Note: When you remotely administer another IIS server, the value of the registry key of the local server is used to determine how the site identification number is generated on the remote server.

Why was it changed?

The reason that Instance ID were changed was that if you are running a Web Farm with multiple physical IIS servers and create a web site on each server with the same description all of the web sites will have the same Instance ID.

Basically if you create a new web site in IIS 6 and then delete the web site and create it again it will be created with the same Instance ID. So if you have multiple physical web servers for a particular web application you can read the log files from the same W3SVCx where x is the instance ID.

I created a web site with a description of XYZ and it produced the Instance ID of 906768 so my log file directory ended up as W3SVC906768

I then deleted this web site and the associated Log File Directory and repeated the exercise.

It indeed did create the same Instance ID of 90768

The following table shows some descriptions and Instance IDs

Description	Instance ID	Comment
A	66	ASCII character for B not A
B	67	ASCII character for C not B
AA	6631	No idea why 31 appears
AB	6632	No idea why 32 appears
AAA	669696	No idea why 96 appears

ClustrMaps - Show a map of where you get your web site visitors....

Chris Crowe — Thu, 06 Apr 2006 23:59:00 GMT

I came across an interesting site today http://clustrmaps.com

See at a glance where your site's visitors are located: instantaneously, even when the numbers are enormous! Visitors don't need to click on anything: just viewing your page is sufficient.

One thumbnail map on your site shows it all: We provide (free) the HTML that gives you a thumbnail map, like the one on the left. When it loads, it increments a counter and shows the locations of all the visitors to your page, cumulatively (even for huge numbers). Clicking on it zooms in to a big world map, and (optionally) lets you zoom in to the continents, as in the example below.

No hidden extras: For light users (under 2500 visitors daily - that is me) the service is free, forever, and stores cumulative totals up to many hundreds of thousands of visitors. Paying users get extra features.

C# application to show the WWW and FTP Sites and their log file directories....

Chris Crowe — Tue, 21 Feb 2006 18:44:00 GMT

I often look at the log files on my web server and with IIS 6 the folders are created with random numbers.

Correction from Tom regarding the “random numbers“:

In fact the log file names are generated from the site name so that in cases where a site is run on multiple servers the site id will be the same on each server. This helps with scripting and stuff. You can configure it in the registry to use the IIS5 type naming format if you want.

More details can be found below:

Microsoft IIS Insider - February 2005
http://www.microsoft.com/technet/community/columns/insider/iisi0205.mspx#ECE

Configure the Web Site Identification Number (IIS 6.0)
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/fb7eb8d1-8862-497b-83d3-eee54a98a2de.mspx

This simple application will display the WWW and FTP sites along with the log file directory. Just compile
it up or download the executable and drop the executable into your c:\windows\system32\logfiles folder and
just dblclick on it when you need to view the sites to folders relationships.

using System;
using System.DirectoryServices;
using System.IO;
using System.Collections;
using System.Windows.Forms;
namespace IISHelpDir
{
    /// 
    /// Summary description for Class1.
    /// 
    class Class1
    {
        /// 
        /// The main entry point for the application.
        /// 
        [STAThread]
        static void Main(string[] args)
        {
            SortedList www = new SortedList();
            SortedList ftp = new SortedList();
            try
            {
                const string FtpServerSchema = "IIsFtpServer"; // Case Sensitive
                const string WebServerSchema = "IIsWebServer"; // Case Sensitive
                string ServerName = "LocalHost";
                DirectoryEntry W3SVC = new DirectoryEntry("IIS://" + ServerName + "/w3svc");
                foreach (DirectoryEntry Site in W3SVC.Children) 
                {
                    if (Site.SchemaClassName == WebServerSchema) 
                    {
                        string LogFilePath = System.IO.Path.Combine(
                            Site.Properties["LogFileDirectory"].Value.ToString(),
                            "W3CSVC"+Site.Name);
                        www.Add(Site.Properties["ServerComment"].Value.ToString(), LogFilePath);
                    }
                }
    
                DirectoryEntry MSFTPSVC = new DirectoryEntry("IIS://" + ServerName + "/msftpsvc");
                foreach (DirectoryEntry Site in MSFTPSVC.Children) 
                {
                    if (Site.SchemaClassName == FtpServerSchema) 
                    {
                        string LogFilePath = System.IO.Path.Combine(
                            Site.Properties["LogFileDirectory"].Value.ToString(), 
                            "MSFTPSVC"+Site.Name);
                        ftp.Add(Site.Properties["ServerComment"].Value.ToString(), LogFilePath);
                    }
                }
                int MaxWidth = 0;
                foreach(string Site in www.Keys)
                {
                    if (Site.Length > MaxWidth)
                        MaxWidth = Site.Length;
                }
                foreach(string Site in ftp.Keys)
                {
                    if (Site.Length > MaxWidth)
                        MaxWidth = Site.Length;
                }
                Console.WriteLine("Site Description".PadRight(MaxWidth)+"  Log File Directory");
                Console.WriteLine("".PadRight(79,'='));
                Console.WriteLine();
                Console.WriteLine("WWW Sites");
                Console.WriteLine("=========");
                foreach(string Site in www.Keys)
                {
                    Console.WriteLine(Site.PadRight(MaxWidth) + "  " + www[Site]);
                }                
                if (ftp.Keys.Count > 0)
                {
                    Console.WriteLine();
                    Console.WriteLine("FTP Sites");
                    Console.WriteLine("=========");
                    foreach(string Site in ftp.Keys)
                    {
                        Console.WriteLine(Site.PadRight(MaxWidth) + "  " + ftp[Site]);
                    }                
                }
            }
                // Catch any errors
            catch (Exception e) 
            {
                Console.WriteLine("Error: " + e.ToString());
            }
            finally
            {
                Console.WriteLine();
                Console.WriteLine("Press enter to close/exit....");
                Console.Read();
            }
        }
    }
}

To download a ZIP file containing the c# source and executable (.Net 1.1) please click here.

Basic Web Site Usage Statistics - SpotCheck

Chris Crowe — Thu, 13 Oct 2005 02:27:00 GMT

Spotcheck is a handy, freeware program that provides basic site usage statistics from IIS log files. You don't have to look at any advertising or give us an email address. Just download and use.

For more details see : http://www.tcpdata.com/sc_dl.shtml

Preventing Log Evasion in IIS

Chris Crowe [IIS MVP] — Sun, 28 Aug 2005 13:24:00 GMT

One of the most important functions a Web site has is the ability to track who is visiting it, where they are coming from, and what they are doing. While logs themselves may not always be the most accurate measurement of what's going on, they do provide a high level overview useful for tracking common user functions and tasks. There are instances when certain types of data aren't logged such as referrers, cookies, user agents, and POST data. Logging can also be used to track abnormal behavior including malicious requests sent by a potential attacker trying to break into your site. These logs can be extremely valuable in identifying if an attack was successful or not, as well as some of the exact commands that an attacker may have executed.

For more details see the full article at http://www.webappsec.org/projects/articles/082905.shtml

IIS - HTTP Status Codes

Chris Crowe — Fri, 26 Aug 2005 01:18:00 GMT

When you are checking your IIS log files you can find a field which defines the status of the request. This status can be very useful when you are trying to diagnose a problem such as a user being denied access to your site.

An example from a W3Extended log file format containing 2 log entries from Windows XP Professional

#Software: Microsoft Internet Information Services 5.1
#Version: 1.0
#Date: 2005-08-26 18:19:49
#Fields: date time c-ip cs-username s-sitename s-computername s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status sc-win32-status sc-bytes cs-bytes time-taken cs-version cs-host
2005-08-26 18:19:49 127.0.0.1 - W3SVC1 CHRIS 127.0.0.1 80 GET /images/ - 302 0 285 586 62 HTTP/1.1 localhost
2005-08-26 18:19:49 127.0.0.1 - W3SVC1 CHRIS 127.0.0.1 80 GET /images/ - 403 5 334 587 16 HTTP/1.1 localhost

From the above log entries we can see we have a status of 302 (Object moved) for the first request, and a status of 403 (Forbidden) for the second request. But we do not know why the user was denied access. In this case I tried to browse an image directory and it did not have directory browsing enabled which should have logged a 403.14 error but IIS 5.1 and earlier do not support storing the sub status code.

Doing something similar with IIS 6 on Windows 2003 Server we get these log file entries.

#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2005-08-26 00:03:26
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes
2005-08-26 18:33:30 W3SVC68783193 SBS2003 192.168.2.2 GET /images - 80 - 192.168.2.1 HTTP/1.1 301 0 0 399 432
2005-08-26 18:33:30 W3SVC68783193 SBS2003 192.168.2.2 GET /images/ - 80 - 192.168.2.1 HTTP/1.1 403 14 5 412 433

In the IIS 6 log file example above you can see that I am logging two status fields sc-status and sc-substatus

This time the first request is returning a status of 301 (Object Moved Permanently) and a sub status of 0 which is not used.
The second request returns a status of 403 (Forbidden) and a sub status of 14 (Directory Listing Denied)

1xx - Informational

These status codes indicate a provisional response. The client should be prepared to receive one or more 1xx responses before receiving a regular response.

100 - Continue.
101 - Switching protocols.

2xx - Success

This class of status codes indicates that the server successfully accepted the client request.

200 - OK. The client request has succeeded.
201 - Created.
202 - Accepted.
203 - Non-authoritative information.
204 - No content.
205 - Reset content.
206 - Partial content.

3xx - Redirection

The client browser must take more action to complete the request. For example, the browser may have to request a different page on the server or repeat the request by using a proxy server.

301 - Moved Permanently
302 - Object moved Temporarily
303 - See Other
304 - Not modified.
307 - Temporary redirect.

4xx - Client Error

An error occurs, and the client appears to be at fault. For example, the client may request a page that does not exist, or the client may not provide valid authentication information.

400 - Bad request.
401 - Access denied. IIS defines a number of different 401 errors that indicate a more specific cause of the error. These specific error codes are displayed in the browser but are not displayed in the IIS log:

401.1 - Logon failed.
401.2 - Logon failed due to server configuration.
401.3 - Unauthorized due to ACL on resource.
401.4 - Authorization failed by filter.
401.5 - Authorization failed by ISAPI/CGI application.
401.7 – Access denied by URL authorization policy on the Web server. This error code is specific to IIS 6.0.

403 - Forbidden. IIS defines a number of different 403 errors that indicate a more specific cause of the error:

403.1 - Execute access forbidden.
403.2 - Read access forbidden.
403.3 - Write access forbidden.
403.4 - SSL required.
403.5 - SSL 128 required.
403.6 - IP address rejected.
403.7 - Client certificate required.
403.8 - Site access denied.
403.9 - Too many users.
403.10 - Invalid configuration.
403.11 - Password change.
403.12 - Mapper denied access.
403.13 - Client certificate revoked.
403.14 - Directory listing denied.
403.15 - Client Access Licenses exceeded.
403.16 - Client certificate is untrusted or invalid.
403.17 - Client certificate has expired or is not yet valid.
403.18 - Cannot execute requested URL in the current application pool. This error code is specific to IIS 6.0.
403.19 - Cannot execute CGIs for the client in this application pool. This error code is specific to IIS 6.0.
403.20 - Passport logon failed. This error code is specific to IIS 6.0.

404 - Not found. 404.0 - (None) – File or directory not found.

404.1 - Web site not accessible on the requested port.
404.2 - Web service extension lockdown policy prevents this request.
404.3 - MIME map policy prevents this request.
404.4 - No Handler (IIS 7)
404.5 - Request Filtering: URL Sequence Denied (IIS 7)
404.6 - Request Filtering: Verb denied (IIS 7)
404.7 - Request Filtering: File extension denied (IIS 7)
404.8 - Request Filtering: Denied by hidden namespace (IIS 7)
404.9 - Denied since hidden file attribute has been set (IIS 7)
404.10 - Request Filtering: Denied because request header is too long (IIS 7)
404.11- Request Filtering: Denied because URL doubled escaping (IIS 7)
404.12 - Request Filtering: Denied because of high bit characters (IIS 7)
404.13 - Request Filtering: Denied because content length too large (IIS 7)
404.14 - Request Filtering: Denied because URL too long (IIS 7)
404.15- Request Filtering: Denied because query string too long (IIS 7)

405 - HTTP verb used to access this page is not allowed (method not allowed.)
406 - Client browser does not accept the MIME type of the requested page.
407 - Proxy authentication required.
412 - Precondition failed.
413 – Request entity too large.
414 - Request-URI too long.
415 – Unsupported media type.
416 – Requested range not satisfiable.
417 – Execution failed.
423 – Locked error.

5xx - Server Error

The server cannot complete the request because it encounters an error.

500 - Internal server error.

500.12 - Application is busy restarting on the Web server.
500.13 - Web server is too busy.
500.15 - Direct requests for Global.asa are not allowed.
500.16 – UNC authorization credentials incorrect. This error code is specific to IIS 6.0.
500.18 – URL authorization store cannot be opened. This error code is specific to IIS 6.0.
500.100 - Internal ASP error.

501 - Header values specify a configuration that is not implemented.
502 - Web server received an invalid response while acting as a gateway or proxy.

502.1 - CGI application timeout.
502.2 - Error in CGI application.

503 - Service unavailable. This error code is specific to IIS 6.0.
504 - Gateway timeout.
505 - HTTP version not supported.

For more details see these resources :

RFC 2616 - Http Status Codes
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
Microsoft KB Article - Common HTTP Status Codes and Their Causes
http://support.microsoft.com/default.aspx?scid=kb;en-us;318380

Log Parser - Different Output Formats available

Chris Crowe [IIS MVP] — Wed, 24 Aug 2005 04:41:00 GMT

In this blog entry we will display the same output in a number of different formats that Log Parser is capable of providing.

Default

Using the default output format the results are displayed inside of the command prompt.

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Output

Notice in the above listing you get a "Press a key..." displayed you can turn this off if you use the -rtp:-1 switch

Chart

Using an output format of a chart you can create nice graphs of log entries

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
into test.gif
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql -view

Output

The output in this case is a file on disk called test.gif. and the -view parameter displays it in a window.

DataGrid

Using an output format of a DataGrid you can view the results inside of a grid which is a lot easier for viewing the results in certain circumstances.

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
into DATAGRID
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Output

LogParser - How to retreive the log filename where a LogFileEntry is from.

Chris Crowe [IIS MVP] — Wed, 24 Aug 2005 03:46:00 GMT

When using Log Parser you may want to include the log file name that the client data was extracted from. If you are using the W3C format you can do this with the LogFilename input field which will return the full path to the log filename that contains the row of data.

Save the data below as distinctclientrequests.sql

SELECT top 25 distinct LogFilename, c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip, LogFilename
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Example Output:

LogFilename	ClientIP	Hits
\\sbs2003\LogFiles\W3SVC68783193\ex050819.log	192.168.2.1	2791
\\sbs2003\LogFiles\W3SVC68783193\ex050809.log	192.168.2.1	2296
\\sbs2003\LogFiles\W3SVC68783193\ex050807.log	218.101.54.21	2262
\\sbs2003\LogFiles\W3SVC68783193\ex050806.log	192.168.2.1	1967
\\sbs2003\LogFiles\W3SVC68783193\ex050811.log	192.168.2.1	1838
\\sbs2003\LogFiles\W3SVC68783193\ex050808.log	218.101.54.21	1744
\\sbs2003\LogFiles\W3SVC68783193\ex050810.log	192.168.2.1	1441
\\sbs2003\LogFiles\W3SVC68783193\ex050804.log	218.101.54.21	1372
\\sbs2003\LogFiles\W3SVC68783193\ex050811.log	218.101.54.21	1243
\\sbs2003\LogFiles\W3SVC68783193\ex050824.log	192.168.2.1	965

If you are wanting to only get the log filename and not the full path you could use the following query:

Save the data below as distinctclientrequests.sql

SELECT top 25 distinct EXTRACT_FILENAME(LogFilename) as LogFile, c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip, LogFile
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Example Output:

LogFile	ClientIP	Hits
ex050819.log	192.168.2.1	2791
ex050809.log	192.168.2.1	2296
ex050807.log	218.101.54.21	2262
ex050806.log	192.168.2.1	1967
ex050811.log	192.168.2.1	1838
ex050808.log	218.101.54.21	1744
ex050810.log	192.168.2.1	1441
ex050804.log	218.101.54.21	1372
ex050811.log	218.101.54.21	1243
ex050824.log	192.168.2.1	965