Posts tagged ‘Analyzing IIS logs with LogParser’

Analyzing IIS logs with LogParser

When users access your server running IIS, IIS logs the information. The logs provide valuable information that you can use to identify any unauthorized attempts to compromise your Web server.
Depending on the amount of traffic to your Web site, the size of your log file (or the number of log files) can consume valuable disk space, memory resources, and CPU cycles. You might need to balance the gathering of detailed data with the need to limit files to a manageable size and number. Logging information in IIS goes beyond the scope of the event logging or performance monitoring features provided by Windows. The IIS logs can include information, such as who has visited your site, what the visitor viewed, and when the information was last viewed.

IIS log file format:

IIS log file format is a fixed (meaning that it cannot be customized) ASCII format. This file format records more information than other log file formats, including basic items, such as the IP address of the user, user name, request date and time, service status code, and number of bytes received. In addition, IIS log file format includes detailed items, such as the elapsed time, number of bytes sent, action (for example, a download carried out by a GET command), and target file. The IIS log file is an easier format to read than the other ASCII formats because the information is separated by commas, while most other ASCII log file formats use spaces for separators. Time is recorded as local time.

IIS log file location:

The IIS logs provide a great deal of information about the activity of a Web application. You can find the IIS logs in

systemroot\System32\LogFiles\W3SVCnumber, where number is the site ID for the Web site.

LogParser:

LogParser is a command line utility. The default behaviour of LogParser is it works like a “data processing pipeline”, by taking an SQL expression on the command line, and outputting the lines containing matches for the SQL expression.

Download LopParser from:
http://www.microsoft.com/downloads/en/details.aspx?familyid=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

W3C Extended Logging Field Definitions


Prefix Meaning
s- Sever actions
c- Client actions
cs- Client-to-server actions.
sc- Server-to-client actions.


Field Appears As Description
Date date The date that the activity occurred.
Time time The time that the activity occurred.
Client IP Address c-ip The IP address of the client that accessed your server.
User Name cs-username The name of the authenticated user who accessed your server. This does not include anonymous users, who are represented by a hyphen (-).
Service Name s-sitename The Internet service and instance number that was accessed by a client.
Server Name s-computername The name of the server on which the log entry was generated.
Server IP Address s-ip The IP address of the server on which the log entry was generated.
Server Port s-port The port number the client is connected to.
Method cs-method The action the client was trying to perform (for example, a GET method).
URI Stem cs-uri-stem The resource accessed; for example, Default.htm.
URI Query cs-uri-query The query, if any, the client was trying to perform.
Protocol Status sc-status The status of the action, in HTTP or FTP terms.
Win32® Status sc-win32-status The status of the action, in terms used by Microsoft Windows®.
Bytes Sent sc-bytes The number of bytes sent by the server.
Bytes Received cs-bytes The number of bytes received by the server.
Time Taken time-taken The duration of time, in milliseconds, that the action consumed.
Protocol Version cs-version The protocol (HTTP, FTP) version used by the client. For HTTP this will be either HTTP 1.0 or HTTP 1.1.
Host cs-host Displays the content of the host header.
User Agent cs(User-Agent) The browser used on the client.
Cookie cs(Cookie) The content of the cookie sent or received, if any.
Referrer cs(Referer) The previous site visited by the user. This site provided a link to the current site.



The following is an example of a record in the extended log format that was produced by the Microsoft Internet Information Server (IIS):
——————————————————————————–
#Software: Microsoft Internet Information Server 6.0
#Version: 1.0
#Date: 2011-05-09 22:48:39
#Fields: date time c-ip cs-username s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-version cs(User-Agent) cs(Cookie) cs(Referrer)

2011-05-09 22:48:39 192.168.1.5 – 173.201.216.31 /GreenBlue.jpg – 200 540 324 157 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+95) USERID=CustomerA;+IMPID=01234 http://www.punebids.com

Procedure:

  1. First note the date and timings of the test run.
  2. Collect the IIS log files of that particular timeframe from the Web Servers (specific to your Web Application) and put it in a single folder
  3. Write query for data collection from logs and run it in LogParser

To find URL Hit count:


1
logparser “SELECT cs-uri-stem AS Url, count(cs-uri-stem) AS Hits FROM ‘C:\Logs\WebServers1-4\*.*’ WHERE date=to_timestamp(2011-05-10,’yyyy-MM-dd’) and time>01:55:00’ and time<02:35:00’ and cs-uri-stem like ‘%asp%’ GROUP BY Url ORDER BY Hits DESC” –i:IISW3C –o:csv >”C:\Logs\CollectedUrlHits.txt



Output:

Url,Hits

/Framework/website/Default.aspx,15678

/isvs/consulting/userprofile.asp,897

/DNA/Common/Portal/ClientHome.aspx,75

/DNA/Common/Clients/Clientlist.aspx,75

/DNA/Common/portal/DNAuserlist.aspx,75

Statistics:

Elements Processed: 245646

Elements output: 5

Execution time: 5.80 seconds

To find the HTTP Error count:


1
logparser “SELECT cs-uri-stem AS Url, count(cs-uri-stem) AS Hits  FROM ‘C:\Logs\WebServers1-4\*.*’ WHERE date=to_timestamp(2011-05-10,’yyyy-MM-dd’) and time>01:55:00’ and time<02:35:00’ and cs-uri-stem like ‘%asp%’ and sc-status=500  GROUP BY Url ORDER BY Hits DESC” –i:IISW3C –o:csv >”C:\Logs\Collected500Errors.txt


Output:

Url,Hits

/Framework/website/Default.aspx,1

/isvs/consulting/userprofile.asp,2

/DNA/Common/Portal/ClientHome.aspx,1

Statistics:

Elements Processed: 245646

Elements output: 3

Execution time: 3.80 seconds





P.S. : Thanks to Rahul D.