Log Parser Studio Scenarios - Real time CAS logging using a checkpoint file.

Much of the time LPS is used to query log files after the fact. You are troubleshooting a potential issue and are simply running queries against IIS logs for example. However, what if you want to run queries against "live" logs and you only want to return results in real time as new entries are logged, ignoring entries that existed in previous runs of the query? This can be done by using a checkpoint file for IIS logs.

This works by pointing LPS to a checkpoint file location. This can be any filename and location you choose, just enter it into the IISW3CLOG format properties and everything else is automatic. The first time a query is run, the checkpoint file is updated with the position of the last row contained in the logs. Any subsequent runs will now BEGIN at the first row created beyond that checkpoint. This means you could have a production server logging thousands of requests per minute and each run of the query will only return results for requests made since the previous run.

One thing you always want to remember about this method is to only use it on live log files that are growing in real time. If you set a checkpoint file and start running queries on static logs the following will occur:

1. The first query returns results.
2. Every subsequent run returns absolutely nothing because no new rows have been added!

I wanted to point this out because on more than one occasion I have set a checkpoint file, forgotten about it then started querying static logs. This makes it appear that your finely crafted query just quit working and has problems and if you don't realize what just happened you'll pull more than your share of hairs out trying to fix a perfectly good query. So remember, using a checkpoint file is a great tool for querying live logs where you need real time results of incoming requests but it is a really bad idea for logs that are static and not being actively logged to.

To set a checkpoint file, click the gear icon to the right of the log type drop down in the query window and enter the file/path for the checkpoint file property. Be aware that only IISW3C logs and possibly one or two others contain a checkpoint file option. Whether this option is available or not depends on the underlying Log Parser 2.2 input format.