IIS - Log Parser

Microsoft releases a new download center for IIS (everything in one place)

Chris Crowe — Thu, 01 Feb 2007 19:36:00 GMT

DownloadCENTER for IIS.net has been released!

The DownloadCENTER at IIS.net, is a community hotspot for discovering, sharing, reviewing and promoting IIS-related solutions in a single place. Dozens of existing downloads, for all versions of IIS – both from Microsoft and the community – are already available in DownloadCENTER today.

This new feature of IIS.net is particularly relevant with the release of IIS7 in Windows Vista. The latest release of Microsoft’s Web server has a completely modular architecture which features over forty pluggable components that can be easily added, removed or even replaced with custom implementations.

This powerful extensibility support is available to both .NET and C/C++ developers. In the future, DownloadCENTER is expected to house a large number of IIS7 extensions submitted by not only the IIS team but the developers and partner ISVs of the IIS community as well.

To learn more about the DownloadCenter, read IIS Product Unit Manager, Bill Staples’ blog post about it or check it out yourself today!

Christchurch .NET Users Group Meeting - June 21st, 2006

Chris Crowe — Fri, 23 Jun 2006 19:29:00 GMT

I had the pleasure to present to the Christchurch .NET Users Group ( http://www.dot.net.nz ) last evening. It was a very cold night with hail, sleet, rain and we had around 25 people attend.

I spoke about three topics in my presentation:

IIS Scripting
IIS Diagnostic Tools
IIS 7

IIS Scripting

In the talk on IIS Scripting I discussed the different providers that were available to manage IIS from a command line script or application. These include ADSI and WMI, Admin Base Objects and the new IIS 7 Managed Provider.

References:

ADSI Provider
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/iissdk/html/965cc820-753b-4ab7-b1c4-8ab43d67e410.asp
WMI Provider
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/iissdk/html/8b7799c1-cebc-4b11-b3d5-34ab7b728077.asp
Admin Base Objects
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/iissdk/html/6f6a0cfa-4f2b-4641-88c5-2a1767f6fbc8.asp
IIS7 Managed Provider - Microsoft.Web.Administration
http://www.iis.net/default.aspx?tabid=2&subtabid=25&i=952

IIS Diagnostic Tools

In the discussion about IIS diagnostic tools I presented details on the following tools from the Debug Diagnostic Toolkit for IIS.

Authentication and Access Control Diagnostics

Authentication and Access Control Diagnostics (Authdiag) Version 1.0 allows you to review, test, and correct problems with Internet Information Services (IIS) authentication and authorization. You can use Authdiag to check settings on Web sites, FTP sites, virtual directories, Web directories, and files. Authdiag can help you troubleshoot the following types of issues:

SSL Diagnostics

It provides a centralized location to display all relevant SSL configuration information. Most of this information is stored in the metabase, which is the main IIS configuration file. Information related to certificates is also stored in the Windows registry.

SSL Diagnostics checks for the correct configuration of SSL objects and settings. These include client and server certificates, ports, private keys, and Web site states.

Administrators can test whether their current server certificate is working properly by temporarily replacing the current certificate with a self-signed certificate. Administrators are able to test their certificates with a single click of the mouse.

Administrators can use SSL Diagnostics to quickly simulate the connection, or handshake, between the server and browser, and review the response from the server. This is very helpful for determining where in the SSL handshake process the connection is breaking down.

Debug Diagnostics

Debug Diagnostics (DebugDiag) 1.0 is a comprehensive tool designed to help IIS administrators or developers determine why a IIS worker process is crashing, hanging, or memory leak. It offers a simple User Interface to build rules for capturing these common problems with web applications and also offers a built-in analysis system

References:

IIS Debug Diagnostic Tool Kit (6 MB)
http://www.iis.net/default.aspx?tabid=3&subtabid=31&g=5&i=895
Microsoft Log Parser 2.2
http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en
www.logparser.com
Fiddler - HTTP Debugging Proxy Server
www.fiddlertool.com

IIS7

In the IIS 7 section of the presentation I discussed some of the benefits of the new server and its new modular architecture, new UI, new extensibility, and diagnostics functionality.

I showed a number of demos of IIS 7 including a custom Basic Authentication module, a custom Directory Browsing Module, Tracing Features, Debugging a crashing application pool with Debug Diag and some features of the new User Interface.

References

The new official IIS Community Home
www.iis.net
Try IIS7 today in your web browser
http://www.microsoftvirtuallabs.com/iisnet/

My presentation was made on the Beta 2 build of Vista Ultimate and most of the tools worked fine on Vista and IIS 7 even though they are not designed to. As far as I know no body left early - a good sign.... IIS can be a dry topic to developers but I hope they learned something useful about the new product and of course scripting and diagnostics that they may not have known previously.

Preventing Log Evasion in IIS

Chris Crowe [IIS MVP] — Mon, 29 Aug 2005 07:24:00 GMT

One of the most important functions a Web site has is the ability to track who is visiting it, where they are coming from, and what they are doing. While logs themselves may not always be the most accurate measurement of what's going on, they do provide a high level overview useful for tracking common user functions and tasks. There are instances when certain types of data aren't logged such as referrers, cookies, user agents, and POST data. Logging can also be used to track abnormal behavior including malicious requests sent by a potential attacker trying to break into your site. These logs can be extremely valuable in identifying if an attack was successful or not, as well as some of the exact commands that an attacker may have executed.

For more details see the full article at http://www.webappsec.org/projects/articles/082905.shtml

Log Parser - Different Output Formats available

Chris Crowe [IIS MVP] — Wed, 24 Aug 2005 22:41:00 GMT

In this blog entry we will display the same output in a number of different formats that Log Parser is capable of providing.

Default

Using the default output format the results are displayed inside of the command prompt.

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Output

Notice in the above listing you get a "Press a key..." displayed you can turn this off if you use the -rtp:-1 switch

Chart

Using an output format of a chart you can create nice graphs of log entries

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
into test.gif
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql -view

Output

The output in this case is a file on disk called test.gif. and the -view parameter displays it in a window.

DataGrid

Using an output format of a DataGrid you can view the results inside of a grid which is a lot easier for viewing the results in certain circumstances.

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
into DATAGRID
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Output

LogParser - How to retreive the log filename where a LogFileEntry is from.

Chris Crowe [IIS MVP] — Wed, 24 Aug 2005 21:46:00 GMT

When using Log Parser you may want to include the log file name that the client data was extracted from. If you are using the W3C format you can do this with the LogFilename input field which will return the full path to the log filename that contains the row of data.

Save the data below as distinctclientrequests.sql

SELECT top 25 distinct LogFilename, c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip, LogFilename
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Example Output:

LogFilename	ClientIP	Hits
\\sbs2003\LogFiles\W3SVC68783193\ex050819.log	192.168.2.1	2791
\\sbs2003\LogFiles\W3SVC68783193\ex050809.log	192.168.2.1	2296
\\sbs2003\LogFiles\W3SVC68783193\ex050807.log	218.101.54.21	2262
\\sbs2003\LogFiles\W3SVC68783193\ex050806.log	192.168.2.1	1967
\\sbs2003\LogFiles\W3SVC68783193\ex050811.log	192.168.2.1	1838
\\sbs2003\LogFiles\W3SVC68783193\ex050808.log	218.101.54.21	1744
\\sbs2003\LogFiles\W3SVC68783193\ex050810.log	192.168.2.1	1441
\\sbs2003\LogFiles\W3SVC68783193\ex050804.log	218.101.54.21	1372
\\sbs2003\LogFiles\W3SVC68783193\ex050811.log	218.101.54.21	1243
\\sbs2003\LogFiles\W3SVC68783193\ex050824.log	192.168.2.1	965

If you are wanting to only get the log filename and not the full path you could use the following query:

Save the data below as distinctclientrequests.sql

SELECT top 25 distinct EXTRACT_FILENAME(LogFilename) as LogFile, c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip, LogFile
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Example Output:

LogFile	ClientIP	Hits
ex050819.log	192.168.2.1	2791
ex050809.log	192.168.2.1	2296
ex050807.log	218.101.54.21	2262
ex050806.log	192.168.2.1	1967
ex050811.log	192.168.2.1	1838
ex050808.log	218.101.54.21	1744
ex050810.log	192.168.2.1	1441
ex050804.log	218.101.54.21	1372
ex050811.log	218.101.54.21	1243
ex050824.log	192.168.2.1	965

Log Parser - Sample - How to return the number of files and their size in a folder and all child folders.

Chris Crowe [IIS MVP] — Thu, 11 Aug 2005 03:34:00 GMT

The Log Parser 2.2 and greater is a very powerful little tool for things that you may not even think about such as enumerating the number of files in a folder and the size that they are consuming.

With these examples we will be using the -i:FS input parameter which means we are getting our data from the File System.

Example 1

The command below will return the total number of files and the size of those files in the c:\temp folder and all child folders.

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM 'c:\temp\*.*' " -i:FS

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files     Size
-----------         ----------
3154               3856346363

Example 2

To restrict the returned details where files start with the letter 'A' you can use this command:

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM c:\temp\*.* where Name like 'A%' " -i:FS

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files       Size
-----------           ----------
98                    336363

Example 3

To restrict the returned details where files that have an extension of 'XML' you can use this command:

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM c:\temp\*.* where EXTRACT_EXTENSION(Name) = 'xml' " -i:FS

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files      Size
-----------          ----------
16                   234374

The 3 examples above have all defaulted to recursively searching all child folders. You can control this with the recurse parameter.

-recurse:-1 Unlimited depth
-recurse:0 Disable recursion
-recurse:5 Recursively search to a maximum depth of 5 folders

Example 4

Return the total XML files and size, recursively searching folders to a depth of two.

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM c:\temp\*.* where EXTRACT_EXTENSION(Name) = 'xml' " -i:FS -recurse:2

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files      Size
-----------          ----------
26                   433224

Example 5

Return the filename, and calculates the MD5 hash of the content of the XML files in the current folder.

LogParser.exe "SELECT Name, HASHMD5_FILE(Path) FROM c:\temp\*.* where EXTRACT_EXTENSION(Name) = 'xml' " -i:FS -recurse:0

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Name        HASHMD5_FILE(Path)
-------------- --------------------------------
1.xml         7F404D2B02690712C4800D03928FE292
2.xml         60F3600467DD164D9D748D3DCB212CE6
3.xml         B72BABBC411445B5BAA144983046E57A
4.xml         893D233CFCD8A07666960303C7F35657

Example 6

Create a chart of the number of files in c:\temp and break it down by file type, and displays it (it also writes a file called chart.gif). ( very powerful )

LogParser.exe "SELECT EXTRACT_EXTENSION(Name), count(Size) into chart.gif FROM c:\temp\*.* group by EXTRACT_EXTENSION(name)" -i:FS -view -charttitle "Count of files by type in c:\temp"

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Freeware - IIS / Tools / Log Files / Microsoft Log Parser 2.2

Chris Crowe [IIS MVP] — Sun, 07 Aug 2005 23:43:00 GMT

Microsoft Log Parser is a very cool little tool that you can use with a SQL query language to render details from a number of different log file formats including:

IIS log files in the NCSA Common, Combined, and Extended Log File Formats
IIS log files in the Microsoft Log File Format.
IIS log files in the W3C Extended Log File Format
IIS log files in the Centralized Binary Log File Format
IIS when configured to log in the ODBC Log Format
Active Directory Objects
Comma, Tab and Space Delimited Text Files
Enterprise Tracing for Windows trace log files (.etl files) and live ETW trace sessions
Windows Event Log and from Event Log backup files (.evt files).
Files and Directories
HTTP Error log files created by the Http.sys driver (IIS 6+ )
NETMON input format parses network capture files (.cap files) captured by the Network Monitor program (or exported from Ethereal)
Registry Values
Generic text files.
URLScan IIS filter log files
W3C Extended Log File Format
XML Files
Your Own Custom Plugins

You tell Log Parser what information you need and how you want it processed.

The results of your query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart.

An example query:

SELECT TOP 10 cs-uri-stem, COUNT(*)
FROM ex040305.log
GROUP BY cs-uri-stem
ORDER BY COUNT(*) DESC

For more details on Log Parser see http://www.microsoft.com/technet/scriptcenter/tools/logparser/default.mspx

The Unoffical Log Parser web site created and maintained by Mike Gunderloy see http://www.logparser.com/

For some additional scripts and code examples for using Log Parser from c# see http://www.logparser.com/Repository.htm

A book has been released called the Log Parser Toolkit - see http://www.syngress.com/catalog/?pid=3110

For a details explanation of how Log Parse works see http://www.microsoft.com/technet/community/columns/profwin/pw0505.mspx

For examples of using the COM interface to Log Parser see http://www.microsoft.com/technet/community/columns/scripts/sg0105.mspx