November 2010 (1)
August 2010 (1)
July 2010 (1)
June 2010 (3)
July 2009 (3)
June 2009 (1)
May 2009 (1)
February 2009 (1)
January 2009 (1)
November 2008 (3)
October 2008 (4)
September 2008 (9)
August 2008 (6)
July 2008 (3)
June 2008 (3)
January 2008 (1)
November 2007 (2)
October 2007 (6)
September 2007 (5)
August 2007 (22)
July 2007 (6)
June 2007 (1)
May 2007 (3)
April 2007 (27)
March 2007 (8)
February 2007 (6)
September 2006 (2)
August 2006 (4)
July 2006 (9)
June 2006 (17)
May 2006 (20)
April 2006 (12)
March 2006 (9)
February 2006 (4)
January 2006 (3)
December 2005 (2)
November 2005 (4)
October 2005 (5)
September 2005 (37)
August 2005 (83)
July 2005 (6)

Active Directory / LDAP (0)
ASP.Net (19)
Blackberry Development (4)
c# (34)
c++ (3)
Code Camp (1)
Excel (1)
Exchange (3)
Front Page 2003 (6)
FTP User Editor (4)
HTML / CSS / DHTML (8)
IIS (146)
IIS - Log Parser (7)
IIS / FTP (12)
IIS / Tools / Administration (42)
IIS / Tools / Authentication (6)
IIS / Tools / Compression (8)
IIS / Tools / Crash & Hang (12)
IIS / Tools / ISAPI Filters (17)
IIS / Tools / Log Files (17)
IIS / Tools / Scripts (28)
IIS / Tools / Security (9)
IIS / Tools / SSL (6)
IIS 7 (3)
Internet Information Server (1)
Me (Chris Crowe) (6)
MIME Types (1)
Misc (72)
Oulook Express (2)
Silverlight (1)
SQL Server (27)
SQL Server CTE (1)
Vista (15)
Vista Gadgets (8)
Visual Studio (11)
Voice over BroadBand (1)
Windows (33)
Windows Powershell (3)
Windows Sharepoint Services (0)
Windows Sharepoint Services (15)
Windows Vista (14)
Wine Cellar (1)
WMI (8)
IIS - Log Parser (7) blog.crowe.co.nz.Models.Category

IIS - Log Parser

Microsoft releases a new download center for IIS (everything in one place)

DownloadCENTER for IIS.net has been released! 

The DownloadCENTER at IIS.net, is a community hotspot for discovering, sharing, reviewing and promoting IIS-related solutions in a single place.  Dozens of existing downloads, for all versions of IIS – both from Microsoft and the community – are already available in DownloadCENTER today. 

This new feature of IIS.net is particularly relevant with the release of IIS7 in Windows Vista.  The latest release of Microsoft’s Web server has a completely modular architecture which features over forty pluggable components that can be easily added, removed or even replaced with custom implementations. 

This powerful extensibility support is available to both .NET and C/C++ developers.  In the future, DownloadCENTER is expected to house a large number of IIS7 extensions submitted by not only the IIS team but the developers and partner ISVs of the IIS community as well.

To learn more about the DownloadCenter, read IIS Product Unit Manager, Bill Staples’ blog post about it or check it out yourself today!

 


Christchurch .NET Users Group Meeting - June 21st, 2006

I had the pleasure to present to the Christchurch .NET Users Group ( http://www.dot.net.nz ) last evening. It was a very cold night with hail, sleet, rain and we had around 25 people attend.

I spoke about three topics in my presentation:

IIS Scripting

In the talk on IIS Scripting I discussed the different providers that were available to manage IIS from a command line script or application. These include ADSI and WMI, Admin Base Objects and the new IIS 7 Managed Provider.

References:

IIS Diagnostic Tools

In the discussion about IIS diagnostic tools I presented details on the following tools from the Debug Diagnostic Toolkit for IIS.

Authentication and Access Control Diagnostics

Authentication and Access Control Diagnostics (Authdiag) Version 1.0 allows you to review, test, and correct problems with Internet Information Services (IIS) authentication and authorization. You can use Authdiag to check settings on Web sites, FTP sites, virtual directories, Web directories, and files. Authdiag can help you troubleshoot the following types of issues:

SSL Diagnostics

It provides a centralized location to display all relevant SSL configuration information. Most of this information is stored in the metabase, which is the main IIS configuration file. Information related to certificates is also stored in the Windows registry.

SSL Diagnostics checks for the correct configuration of SSL objects and settings. These include client and server certificates, ports, private keys, and Web site states.

Administrators can test whether their current server certificate is working properly by temporarily replacing the current certificate with a self-signed certificate. Administrators are able to test their certificates with a single click of the mouse.

Administrators can use SSL Diagnostics to quickly simulate the connection, or handshake, between the server and browser, and review the response from the server. This is very helpful for determining where in the SSL handshake process the connection is breaking down.

Debug Diagnostics

Debug Diagnostics (DebugDiag) 1.0 is a comprehensive tool designed to help IIS administrators or developers determine why a IIS worker process is crashing, hanging, or memory leak. It offers a simple User Interface to build rules for capturing these common problems with web applications and also offers a built-in analysis system

References:

IIS7

In the IIS 7 section of the presentation I discussed some of the benefits of the new server and its new modular architecture, new UI, new extensibility, and diagnostics functionality.

I showed a number of demos of IIS 7 including a custom Basic Authentication module, a custom Directory Browsing Module, Tracing Features, Debugging a crashing application pool with Debug Diag and some features of the new User Interface.

References


My presentation was made on the Beta 2 build of Vista Ultimate and most of the tools worked fine on Vista and IIS 7 even though they are not designed to. As far as I know no body left early - a good sign.... IIS can be a dry topic to developers but I hope they learned something useful about the new product and of course scripting and diagnostics that they may not have known previously.


Preventing Log Evasion in IIS

One of the most important functions a Web site has is the ability to track who is visiting it, where they are coming from, and what they are doing. While logs themselves may not always be the most accurate measurement of what's going on, they do provide a high level overview useful for tracking common user functions and tasks. There are instances when certain types of data aren't logged such as referrers, cookies, user agents, and POST data. Logging can also be used to track abnormal behavior including malicious requests sent by a potential attacker trying to break into your site. These logs can be extremely valuable in identifying if an attack was successful or not, as well as some of the exact commands that an attacker may have executed.

For more details see the full article at http://www.webappsec.org/projects/articles/082905.shtml


Log Parser - Different Output Formats available

In this blog entry we will display the same output in a number of different formats that Log Parser is capable of providing.

Default

Using the default output format the results are displayed inside of the command prompt.

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc


Command Line

LogParser.exe file:distinctclientrequests.sql

Output

Notice in the above listing you get a "Press a key..." displayed you can turn this off if you use the -rtp:-1 switch

Chart

Using an output format of a chart you can create nice graphs of log entries

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
into test.gif
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc


Command Line

LogParser.exe file:distinctclientrequests.sql -view

Output

The output in this case is a file on disk called test.gif. and the -view parameter displays it in a window.


 

DataGrid

Using an output format of a DataGrid you can view the results inside of a grid which is a lot easier for viewing the results in certain circumstances.

SELECT top 25 distinct c-ip as ClientIP, Count(*) as Hits
into DATAGRID
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip
order by Hits, c-ip desc


Command Line

LogParser.exe file:distinctclientrequests.sql

Output


LogParser - How to retreive the log filename where a LogFileEntry is from.

When using Log Parser you may want to include the log file name that the client data was extracted from. If you are using the W3C format you can do this with the LogFilename input field which will return the full path to the log filename that contains the row of data.

Save the data below as distinctclientrequests.sql

SELECT top 25 distinct LogFilename, c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip, LogFilename
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Example Output:

LogFilename ClientIP Hits
\\sbs2003\LogFiles\W3SVC68783193\ex050819.log 192.168.2.1 2791
\\sbs2003\LogFiles\W3SVC68783193\ex050809.log 192.168.2.1 2296
\\sbs2003\LogFiles\W3SVC68783193\ex050807.log 218.101.54.21 2262
\\sbs2003\LogFiles\W3SVC68783193\ex050806.log 192.168.2.1 1967
\\sbs2003\LogFiles\W3SVC68783193\ex050811.log 192.168.2.1 1838
\\sbs2003\LogFiles\W3SVC68783193\ex050808.log 218.101.54.21 1744
\\sbs2003\LogFiles\W3SVC68783193\ex050810.log 192.168.2.1 1441
\\sbs2003\LogFiles\W3SVC68783193\ex050804.log 218.101.54.21 1372
\\sbs2003\LogFiles\W3SVC68783193\ex050811.log 218.101.54.21 1243
\\sbs2003\LogFiles\W3SVC68783193\ex050824.log 192.168.2.1 965

If you are wanting to only get the log filename and not the full path you could use the following query:

Save the data below as distinctclientrequests.sql

SELECT top 25 distinct EXTRACT_FILENAME(LogFilename) as LogFile, c-ip as ClientIP, Count(*) as Hits
FROM \\sbs2003\LogFiles\W3SVC68783193\ex0508*.log
group by c-ip, LogFile
order by Hits, c-ip desc

Command Line

LogParser.exe file:distinctclientrequests.sql

Example Output:

LogFile ClientIP Hits
ex050819.log 192.168.2.1 2791
ex050809.log 192.168.2.1 2296
ex050807.log 218.101.54.21 2262
ex050806.log 192.168.2.1 1967
ex050811.log 192.168.2.1 1838
ex050808.log 218.101.54.21 1744
ex050810.log 192.168.2.1 1441
ex050804.log 218.101.54.21 1372
ex050811.log 218.101.54.21 1243
ex050824.log 192.168.2.1 965


Log Parser - Sample - How to return the number of files and their size in a folder and all child folders.

The Log Parser 2.2 and greater is a very powerful little tool for things that you may not even think about such as enumerating the number of files in a folder and the size that they are consuming.

With these examples we will be using the -i:FS input parameter which means we are getting our data from the File System.

Example 1

The command below will return the total number of files and the size of those files in the c:\temp folder and all child folders.

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM 'c:\temp\*.*' " -i:FS
 
Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files     Size
-----------         ----------
3154               3856346363
 

Example 2

To restrict the returned details where files start with the letter 'A' you can use this command:

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM c:\temp\*.* where Name like 'A%' " -i:FS

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files       Size
-----------           ----------
98                    336363

Example 3

To restrict the returned details where files that have an extension of 'XML' you can use this command:

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM c:\temp\*.* where EXTRACT_EXTENSION(Name) = 'xml' " -i:FS

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files      Size
-----------          ----------
16                   234374 
 

The 3 examples above have all defaulted to recursively searching all child folders. You can control this with the recurse parameter.

  • -recurse:-1     Unlimited depth
  • -recurse:0       Disable recursion
  • -recurse:5       Recursively search to a maximum depth of 5 folders

Example 4

Return the total XML files and size, recursively searching folders to a depth of two.

LogParser.exe "SELECT count(Size) as [Total Files], sum(Size) as Size FROM c:\temp\*.* where EXTRACT_EXTENSION(Name) = 'xml' " -i:FS -recurse:2

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Total Files      Size
-----------          ----------
26                   433224 
 

Example 5

Return the filename, and calculates the MD5 hash of the content of the XML files in the current folder.

LogParser.exe "SELECT Name, HASHMD5_FILE(Path) FROM c:\temp\*.* where EXTRACT_EXTENSION(Name) = 'xml' " -i:FS -recurse:0

Note: The command line above should be entered on a single line.

The resulting output is similar to this:

Name        HASHMD5_FILE(Path)
-------------- --------------------------------
1.xml         7F404D2B02690712C4800D03928FE292
2.xml         60F3600467DD164D9D748D3DCB212CE6
3.xml         B72BABBC411445B5BAA144983046E57A
4.xml         893D233CFCD8A07666960303C7F35657
 

Example 6

Create a chart of the number of files in c:\temp and break it down by file type, and displays it (it also writes a file called chart.gif). ( very powerful )

LogParser.exe "SELECT EXTRACT_EXTENSION(Name), count(Size) into chart.gif FROM c:\temp\*.* group by EXTRACT_EXTENSION(name)" -i:FS -view -charttitle "Count of files by type in c:\temp"

Note: The command line above should be entered on a single line.

The resulting output is similar to this:


Freeware - IIS / Tools / Log Files / Microsoft Log Parser 2.2

Microsoft Log Parser is a very cool little tool that you can use with a SQL query language to render details from a number of different log file formats including:

  • IIS log files in the NCSA Common, Combined, and Extended Log File Formats
  • IIS log files in the Microsoft Log File Format.
  • IIS log files in the W3C Extended Log File Format
  • IIS log files in the Centralized Binary Log File Format
  • IIS when configured to log in the ODBC Log Format

  • Active Directory Objects
  • Comma, Tab and Space Delimited Text Files
  • Enterprise Tracing for Windows trace log files (.etl files) and live ETW trace sessions
  • Windows Event Log and from Event Log backup files (.evt files).
  • Files and Directories
  • HTTP Error log files created by the Http.sys driver (IIS 6+ )
  • NETMON input format parses network capture files (.cap files) captured by the Network Monitor program (or exported from Ethereal)
  • Registry Values
  • Generic text files.
  • URLScan IIS filter log files
  • W3C Extended Log File Format
  • XML Files

  • Your Own Custom Plugins

You tell Log Parser what information you need and how you want it processed.

The results of your query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart.  

An example query:

SELECT TOP 10 cs-uri-stem, COUNT(*)
FROM ex040305.log
GROUP BY cs-uri-stem
ORDER BY COUNT(*) DESC

For more details on Log Parser see http://www.microsoft.com/technet/scriptcenter/tools/logparser/default.mspx

The Unoffical Log Parser web site created and maintained by Mike Gunderloy see http://www.logparser.com/

For some additional scripts and code examples for using Log Parser from c# see http://www.logparser.com/Repository.htm

A book has been released called the Log Parser Toolkit - see http://www.syngress.com/catalog/?pid=3110

For a details explanation of how Log Parse works see http://www.microsoft.com/technet/community/columns/profwin/pw0505.mspx

For examples of using the COM interface to Log Parser see http://www.microsoft.com/technet/community/columns/scripts/sg0105.mspx