Analyzing IIS logs with LogParser
When users access your server running IIS, IIS logs the information. The logs provide valuable information that you can use to identify any unauthorized attempts to compromise your Web server.
Depending on the amount of traffic to your Web site, the size of your log file (or the number of log files) can consume valuable disk space, memory resources, and CPU cycles. You might need to balance the gathering of detailed data with the need to limit files to a manageable size and number. Logging information in IIS goes beyond the scope of the event logging or performance monitoring features provided by Windows. The IIS logs can include information, such as who has visited your site, what the visitor viewed, and when the information was last viewed.
IIS log file format:
IIS log file format is a fixed (meaning that it cannot be customized) ASCII format. This file format records more information than other log file formats, including basic items, such as the IP address of the user, user name, request date and time, service status code, and number of bytes received. In addition, IIS log file format includes detailed items, such as the elapsed time, number of bytes sent, action (for example, a download carried out by a GET command), and target file. The IIS log file is an easier format to read than the other ASCII formats because the information is separated by commas, while most other ASCII log file formats use spaces for separators. Time is recorded as local time.
IIS log file location:
The IIS logs provide a great deal of information about the activity of a Web application. You can find the IIS logs in
systemroot\System32\LogFiles\W3SVCnumber, where number is the site ID for the Web site.
LogParser:
LogParser is a command line utility. The default behaviour of LogParser is it works like a “data processing pipeline”, by taking an SQL expression on the command line, and outputting the lines containing matches for the SQL expression.
Download LopParser from:
http://www.microsoft.com/downloads/en/details.aspx?familyid=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en
W3C Extended Logging Field Definitions
| Prefix | Meaning |
| s- | Sever actions |
| c- | Client actions |
| cs- | Client-to-server actions. |
| sc- | Server-to-client actions. |
| Field | Appears As | Description |
| Date | date | The date that the activity occurred. |
| Time | time | The time that the activity occurred. |
| Client IP Address | c-ip | The IP address of the client that accessed your server. |
| User Name | cs-username | The name of the authenticated user who accessed your server. This does not include anonymous users, who are represented by a hyphen (-). |
| Service Name | s-sitename | The Internet service and instance number that was accessed by a client. |
| Server Name | s-computername | The name of the server on which the log entry was generated. |
| Server IP Address | s-ip | The IP address of the server on which the log entry was generated. |
| Server Port | s-port | The port number the client is connected to. |
| Method | cs-method | The action the client was trying to perform (for example, a GET method). |
| URI Stem | cs-uri-stem | The resource accessed; for example, Default.htm. |
| URI Query | cs-uri-query | The query, if any, the client was trying to perform. |
| Protocol Status | sc-status | The status of the action, in HTTP or FTP terms. |
| Win32® Status | sc-win32-status | The status of the action, in terms used by Microsoft Windows®. |
| Bytes Sent | sc-bytes | The number of bytes sent by the server. |
| Bytes Received | cs-bytes | The number of bytes received by the server. |
| Time Taken | time-taken | The duration of time, in milliseconds, that the action consumed. |
| Protocol Version | cs-version | The protocol (HTTP, FTP) version used by the client. For HTTP this will be either HTTP 1.0 or HTTP 1.1. |
| Host | cs-host | Displays the content of the host header. |
| User Agent | cs(User-Agent) | The browser used on the client. |
| Cookie | cs(Cookie) | The content of the cookie sent or received, if any. |
| Referrer | cs(Referer) | The previous site visited by the user. This site provided a link to the current site. |
The following is an example of a record in the extended log format that was produced by the Microsoft Internet Information Server (IIS):
——————————————————————————–
#Software: Microsoft Internet Information Server 6.0
#Version: 1.0
#Date: 2011-05-09 22:48:39
#Fields: date time c-ip cs-username s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-version cs(User-Agent) cs(Cookie) cs(Referrer)
2011-05-09 22:48:39 192.168.1.5 – 173.201.216.31 /GreenBlue.jpg – 200 540 324 157 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+95) USERID=CustomerA;+IMPID=01234 http://www.punebids.com
Procedure:
- First note the date and timings of the test run.
- Collect the IIS log files of that particular timeframe from the Web Servers (specific to your Web Application) and put it in a single folder
- Write query for data collection from logs and run it in LogParser
To find URL Hit count:
1 | logparser “SELECT cs-uri-stem AS Url, count(cs-uri-stem) AS Hits FROM ‘C:\Logs\WebServers1-4\*.*’ WHERE date=to_timestamp(‘2011-05-10’,’yyyy-MM-dd’) and time>’01:55:00’ and time<’02:35:00’ and cs-uri-stem like ‘%asp%’ GROUP BY Url ORDER BY Hits DESC” –i:IISW3C –o:csv >”C:\Logs\CollectedUrlHits.txt” |
Output:
Url,Hits
/Framework/website/Default.aspx,15678
/isvs/consulting/userprofile.asp,897
/DNA/Common/Portal/ClientHome.aspx,75
/DNA/Common/Clients/Clientlist.aspx,75
/DNA/Common/portal/DNAuserlist.aspx,75
Statistics:
Elements Processed: 245646
Elements output: 5
Execution time: 5.80 seconds
To find the HTTP Error count:
1 | logparser “SELECT cs-uri-stem AS Url, count(cs-uri-stem) AS Hits FROM ‘C:\Logs\WebServers1-4\*.*’ WHERE date=to_timestamp(‘2011-05-10’,’yyyy-MM-dd’) and time>’01:55:00’ and time<’02:35:00’ and cs-uri-stem like ‘%asp%’ and sc-status=500 GROUP BY Url ORDER BY Hits DESC” –i:IISW3C –o:csv >”C:\Logs\Collected500Errors.txt” |
Output:
Url,Hits
/Framework/website/Default.aspx,1
/isvs/consulting/userprofile.asp,2
/DNA/Common/Portal/ClientHome.aspx,1
Statistics:
Elements Processed: 245646
Elements output: 3
Execution time: 3.80 seconds
P.S. : Thanks to Rahul D.