Analyzing IIS logs with LogParser

When users access your server running IIS, IIS logs the information. The logs provide valuable information that you can use to identify any unauthorized attempts to compromise your Web server.
Depending on the amount of traffic to your Web site, the size of your log file (or the number of log files) can consume valuable disk space, memory resources, and CPU cycles. You might need to balance the gathering of detailed data with the need to limit files to a manageable size and number. Logging information in IIS goes beyond the scope of the event logging or performance monitoring features provided by Windows. The IIS logs can include information, such as who has visited your site, what the visitor viewed, and when the information was last viewed.

IIS log file format:

IIS log file format is a fixed (meaning that it cannot be customized) ASCII format. This file format records more information than other log file formats, including basic items, such as the IP address of the user, user name, request date and time, service status code, and number of bytes received. In addition, IIS log file format includes detailed items, such as the elapsed time, number of bytes sent, action (for example, a download carried out by a GET command), and target file. The IIS log file is an easier format to read than the other ASCII formats because the information is separated by commas, while most other ASCII log file formats use spaces for separators. Time is recorded as local time.

IIS log file location:

The IIS logs provide a great deal of information about the activity of a Web application. You can find the IIS logs in

systemroot\System32\LogFiles\W3SVCnumber, where number is the site ID for the Web site.

LogParser:

LogParser is a command line utility. The default behaviour of LogParser is it works like a “data processing pipeline”, by taking an SQL expression on the command line, and outputting the lines containing matches for the SQL expression.

Download LopParser from:
http://www.microsoft.com/downloads/en/details.aspx?familyid=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

W3C Extended Logging Field Definitions


Prefix Meaning
s- Sever actions
c- Client actions
cs- Client-to-server actions.
sc- Server-to-client actions.


Field Appears As Description
Date date The date that the activity occurred.
Time time The time that the activity occurred.
Client IP Address c-ip The IP address of the client that accessed your server.
User Name cs-username The name of the authenticated user who accessed your server. This does not include anonymous users, who are represented by a hyphen (-).
Service Name s-sitename The Internet service and instance number that was accessed by a client.
Server Name s-computername The name of the server on which the log entry was generated.
Server IP Address s-ip The IP address of the server on which the log entry was generated.
Server Port s-port The port number the client is connected to.
Method cs-method The action the client was trying to perform (for example, a GET method).
URI Stem cs-uri-stem The resource accessed; for example, Default.htm.
URI Query cs-uri-query The query, if any, the client was trying to perform.
Protocol Status sc-status The status of the action, in HTTP or FTP terms.
Win32® Status sc-win32-status The status of the action, in terms used by Microsoft Windows®.
Bytes Sent sc-bytes The number of bytes sent by the server.
Bytes Received cs-bytes The number of bytes received by the server.
Time Taken time-taken The duration of time, in milliseconds, that the action consumed.
Protocol Version cs-version The protocol (HTTP, FTP) version used by the client. For HTTP this will be either HTTP 1.0 or HTTP 1.1.
Host cs-host Displays the content of the host header.
User Agent cs(User-Agent) The browser used on the client.
Cookie cs(Cookie) The content of the cookie sent or received, if any.
Referrer cs(Referer) The previous site visited by the user. This site provided a link to the current site.



The following is an example of a record in the extended log format that was produced by the Microsoft Internet Information Server (IIS):
——————————————————————————–
#Software: Microsoft Internet Information Server 6.0
#Version: 1.0
#Date: 2011-05-09 22:48:39
#Fields: date time c-ip cs-username s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-version cs(User-Agent) cs(Cookie) cs(Referrer)

2011-05-09 22:48:39 192.168.1.5 – 173.201.216.31 /GreenBlue.jpg – 200 540 324 157 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+95) USERID=CustomerA;+IMPID=01234 http://www.punebids.com

Procedure:

  1. First note the date and timings of the test run.
  2. Collect the IIS log files of that particular timeframe from the Web Servers (specific to your Web Application) and put it in a single folder
  3. Write query for data collection from logs and run it in LogParser

To find URL Hit count:


1
logparser “SELECT cs-uri-stem AS Url, count(cs-uri-stem) AS Hits FROM ‘C:\Logs\WebServers1-4\*.*’ WHERE date=to_timestamp(2011-05-10,’yyyy-MM-dd’) and time>01:55:00’ and time<02:35:00’ and cs-uri-stem like ‘%asp%’ GROUP BY Url ORDER BY Hits DESC” –i:IISW3C –o:csv >”C:\Logs\CollectedUrlHits.txt



Output:

Url,Hits

/Framework/website/Default.aspx,15678

/isvs/consulting/userprofile.asp,897

/DNA/Common/Portal/ClientHome.aspx,75

/DNA/Common/Clients/Clientlist.aspx,75

/DNA/Common/portal/DNAuserlist.aspx,75

Statistics:

Elements Processed: 245646

Elements output: 5

Execution time: 5.80 seconds

To find the HTTP Error count:


1
logparser “SELECT cs-uri-stem AS Url, count(cs-uri-stem) AS Hits  FROM ‘C:\Logs\WebServers1-4\*.*’ WHERE date=to_timestamp(2011-05-10,’yyyy-MM-dd’) and time>01:55:00’ and time<02:35:00’ and cs-uri-stem like ‘%asp%’ and sc-status=500  GROUP BY Url ORDER BY Hits DESC” –i:IISW3C –o:csv >”C:\Logs\Collected500Errors.txt


Output:

Url,Hits

/Framework/website/Default.aspx,1

/isvs/consulting/userprofile.asp,2

/DNA/Common/Portal/ClientHome.aspx,1

Statistics:

Elements Processed: 245646

Elements output: 3

Execution time: 3.80 seconds





P.S. : Thanks to Rahul D.

Managing Application Pools

When you run IIS 6.0 in worker process isolation mode, you can separate different Web applications and Web sites into groups known as application pools. An application pool is a group of one or more URLs that are served by a worker process or set of worker processes. Any Web directory or virtual directory can be assigned to an application pool.

A worker process is user-mode code whose role is to process requests, such as processing requests to return a static page, invoking an ISAPI extension or filter, or running a Common Gateway Interface (CGI) handler.

Every application within an application pool shares the same worker process. Because each worker process operates as a separate instance of the worker process executable, W3wp.exe, the worker process that services one application pool is separated from the worker process that services another. Each separate worker process provides a process boundary so that when an application is assigned to one application pool, problems in other application pools do not affect the application. This ensures that if a worker process fails, it does not affect the applications running in other application pools.

Use multiple application pools when you want to help ensure that applications and Web sites are confidential and secure. For example, an enterprise organization might place its human resources Web site and its finance Web site on the same server, but in different application pools. Likewise, an ISP that hosts Web sites and applications for competing companies might run each company’s Web services on the same server, but in different application pools. Using different application pools to isolate applications helps prevent one customer from accessing, changing, or using confidential information from another customer’s site.

Application Pool Actions:

To manage Application Pools go to IIS Manager and click on application pool which you want to configure for recycling.

Tab and their descriptions:

Element Name Description
Add Application Pool Opens the Add Application Pool dialog box from which you can add an application pool to the Web server.
Set Application Pool Defaults Opens the Application Pool Defaults dialog box from which you can set default values that apply to all application pools that you add to the Web server.
Start Starts the selected application pool.
Stop Stops the selected application pool. This causes the Windows Process Activation Service (WAS) to shut down all running worker processes serving that application pool. An administrator must restart a stopped application pool or else requests made to applications in that application pool will receive HTTP 503-Service Unavailable errors.
Recycle Stops and restarts the selected application pool. Restarting an application pool causes the application pool to be temporarily unavailable until the restart is complete.
Basic Settings Opens the Edit Application Pool dialog box from which you can edit the settings that were specified when the application pool was created. This action is available only when an item is selected from the list on the feature page.
Recycling Opens the Edit Application Pool Recycling Settings wizard from which you can specify conditions under which to recycle an application pool and configure how recycling events are logged.
Advanced Settings Opens the Advanced Settings dialog box from which you can configure advanced settings for the selected application pool.
Rename Enables the Name field of the selected application pool so that you can rename the application pool.
Remove Removes the item that is selected from the list on the feature page.
View Applications Opens the Applications feature page from which you can view the applications that belong to the selected application pool.

Edit Application Pool Recycling Settings:

Use the Recycling Conditions page of the Edit Application Pool Recycling Settings Wizard to configure IIS to periodically restart worker processes in an application pool. This can help you to recover valuable system resources and to better manage faulty worker processes.

Tab and their descriptions:

Element Name Description
Regular time intervals (in minutes) Select this option to specify a time interval, in minutes, at which you want IIS to recycle the worker process. You might choose this option if you have an application that causes problems when it runs for an extended time. Based on what you know about the application, you should set the value to be less than the length of time elapsed before application failure.
Fixed number of requests Select this option to specify the number of requests after which you want IIS to recycle the worker process. You might choose this option if you have an application that causes problems after reaching a certain number of requests. Based on what you know about the application, you should configure the value to be less than the number of requests processed before application failure.
Specific time(s) Select this option to specify a time or times at which you want IIS to recycle the worker process in a 24-hour period. For example, to recycle a worker process at 4:30 A.M. and 4:30 P.M., enter 4:30 AM, 4:30 PM. The time that you specify uses the local time on the Web server. You might choose this option if you have an application that causes problems when it runs for an extended time and you want to recycle the application pool at a specific time, such as a time that is late at night or early in the morning, to avoid a negative impact on users. Based on what you know about the application, you should set the interval to be frequent enough to prevent application failure.
Virtual memory usage (in KB) Select this option to specify the maximum number of kilobytes of your system’s common virtual memory that can be used by a worker process before that process is recycled. You might choose this option when you notice a steady increase in the virtual memory used on your server. This might indicate that an application reserves memory multiple times, which fragments the memory heap. Entering too high a value can severely decrease system performance. At first, you should set the virtual memory threshold to be less than 70 percent of available virtual memory, and then adjust the setting if you have to.
Private memory usage (in KB) Select this option to specify the maximum number of kilobytes of privately allocated system physical memory that can be used by a worker process before the process is recycled. You might choose this option when you have an application that leaks memory. Entering too high a value can severely decrease system performance. At first, you should set this value to be less than 60 percent of the available physical memory on the server, and then adjust this setting if you have to.

Recycling Events to Log:

Use the Recycling Events to Log page of the Edit Application Pool Recycling Settings Wizard to configure IIS to log an event when a worker process is recycled.

You can configure IIS to log information for recycling events that you configure, such as at a fixed interval or for recycling events that occur at runtime, such as when an ISAPI declares itself unhealthy.

Tab and their descriptions:

Element Name Description
Regular time intervals Select this option to log an event when a worker process is recycled at a specified time interval. This option is available only when the Regular time intervals (in minutes) option is selected and a time interval is specified on the previous wizard page.
Virtual memory usage Select this option to log an event when a worker process is recycled after using a specified amount of virtual memory. This option is available only when the Virtual memory usage (in KB) option is selected and a number of kilobytes is specified on the previous wizard page.
Number of requests Select this option to log an event when a worker process is recycled after reaching a specified number of requests. This option is available only when the Fixed number of requests option is selected and a number of requests is specified on the previous wizard page.
Scheduled time(s) Select this option to log an event when a worker process is recycled at a specified time. This option is available only when the Specific time(s) option is selected and a time is specified on the previous wizard page.
Private memory usage Select this option to log an event when a worker process is recycled after using a specified amount of physical memory. This option is available only when the Private memory usage (in KB) option is selected and a number of kilobytes is specified on the previous wizard page.
On-demand Select this option to log an event when you recycle a worker process by using IIS Manager or Appcmd.exe to correct a problem.
Configuration changes Select this option to log an event when a change to configuration causes the application pool to recycle.
Unhealthy ISAPI Select this option to log an event when an ISAPI extension reports to the worker process that it is unhealthy.





P.S. : Thanks to Kranti.