Linking your pivot collections the fun and easy way
In
my previous post, I added to my series of entries on making sense of your ASP.NET
event log error messages. Note that this is entry #4 in this series. The
previous three entries can be found here:
-
Part 1: Parsing
ASP.NET event log error messages for fun and profit
Part 2: [Don’t<br>guess when it comes to performance…](https://www.samuraiprogrammer.com/blog/ct.ashx?id=a7f8712d-020c-4cd3-acf3-8342ddc1e629&url=http%3a%2f%2fwww.samuraiprogrammer.com%2fblog%2f2010%2f09%2f19%2fDontGuessWhenItComesToPerformanceaRegExStory.aspx)
In that last post, I walked through the PAuthorLib.dll and showed you how to crawl
through your event log error messages and create a pivot collection. The result
of that initial effort was a nice view into our events:
While this certainly got the job done and is a very powerful and
compelling view into our data, we need to realize that as our data grow, the amount
of entries in our linked collection is limited. From
the Developer Overview, we see that the maximum number of items we should have
in a single collection is 3,000:
So, while a simple collection will get the job done for your smaller amounts of data,
you will really run into some challenges with your larger datasets like our ASP.NET
event log error messages. To combat this limitation you can create what’s called
a Linked Collection. The idea is that it’s just a way for you to link together
related collections in order to provide a seamless experience for your users.
In our case, a natural break for our collections will be based upon the exception
type with a summary collection and then a separate collection for each exception type.
If I were to draw this out:
Event Log Header Collection Source
The idea behind this structure is that the Exception summary would simply link to
each of these exception collections. First, we’ll create a colleciton source
for our exception summary. As
in our previous collection source (in my last blog post), we inherit from the
AbstractCollectionSource class and use the LoadHeaderData() method to add our facet
categories to the collection. In this case, we’ll create two categories – the
number of Occurrences and the Urls where the exception occurred. Another difference
is that we are going to pass the already parsed collection of messages into the constructor.
The reason for that is so we don’t have to repeat the parsing of the event log messages
multiple times.
class EventLogHeaderCollectionSource : AbstractCollectionSource
{
private IEnumerable<EventLogMessage> m_messages = null;
public EventLogHeaderCollectionSource(IEnumerable<EventLogMessage> messages,
string inputFile)
: base(inputFile)
{
m_messages = messages;
}
#region Facets
private const string OCCURRENCES = "Occurrences";
private const string URLS = "Urls";
#endregion
protected override void LoadHeaderData()
{
this.CachedCollectionData.FacetCategories.
Add(new PivotFacetCategory(OCCURRENCES, PivotFacetType.Number));
this.CachedCollectionData.FacetCategories.
Add(new PivotFacetCategory(URLS, PivotFacetType.String));
this.CachedCollectionData.Name =
"ASP.NET Error Messages - Summary";
this.CachedCollectionData.Copyright =
new PivotLink("Source", "https://www.samuraiprogrammer.com");
}
}
Then, in the LoadItems() method, we provide the logic to generate the PivotItem collection.
The one key item to make note of is the use of the Href property
of the PivotItem object. This is where we specify the collection we wish to
link to this item. Since each of the PivotItems will be a summary of the number
of each exception type – we’ll name the sub-collections by its exception type.
For example, NullReferenceException.cxml, SqlException.cxml, etc.
protected override IEnumerable<PivotItem> LoadItems()
{
var results = from log in m_messages
group log by log.Exceptiontype into l
orderby l.Count() descending, l.Key
select new
{
ExceptionType = l.Key,
ExceptionCount = l.Count()
};
int index = 0;
foreach (var result in results)
{
PivotItem item = new PivotItem(index.ToString(), this);
item.Name = result.ExceptionType;
item.Description = "# of Exceptions: " + result.ExceptionCount.ToString();
item.AddFacetValues(OCCURRENCES, result.ExceptionCount);
item.Href = result.ExceptionType + ".cxml";
...
index++;
yield return item;
}
yield break;
}
Event Log Collection Source Redux
Previously, when we generated the pivot collections, we were outputting all of the
records into a single collection. Now that we are generating a collection for
each exception type, we will need to put a filter in our exception collection and
then incorporate that filter into our item generation. Other than that, the
code we wrote previously remains largely unchanged, so I left the majority of it out
and only included the snippets that we care about below.
class EventLogCollectionSource : AbstractCollectionSource
{
private IEnumerable<EventLogMessage> m_messages = null;
private string m_exceptionType = string.Empty;
public EventLogCollectionSource(
IEnumerable<EventLogMessage> messages,
string exceptionType,
string path)
: base(path)
{
m_messages = messages;
m_exceptionType = exceptionType;
}
protected override void LoadHeaderData()
{
...
this.CachedCollectionData.Name =
string.Format("{0} Error Messages", m_exceptionType);
...
}
protected override IEnumerable<PivotItem> LoadItems()
{
var results = (from message in m_messages
where message.Exceptiontype == m_exceptionType
select message);
int index = 0;
foreach (EventLogMessage message in results)
{
PivotItem item =
new PivotItem(index.ToString(), this);
item.Name = message.Exceptiontype;
item.Description = message.Exceptionmessage;
...
index++;
yield return item;
}
yield break;
}
}
Generate and test the collection
Then, the only thing we have left to do is generate and test our linked collections.
I won’t go into a lengthy explanation of how we generate the collections because I
did that in the last blog entry. I will show the broad strokes required
to tie this all together, though:
// Load the raw messages into a collection
IEnumerable<EventLogMessage> messages =
LoadEventLogMessages(inputFile).ToList();
// Generate summary pivot collection
EventLogHeaderCollectionSource sourceSummary =
new EventLogHeaderCollectionSource(messages, inputFile);
...
summaryTargetFilter1.Write(sourceSummaryFilter1);
// Get the aggregate results so we know the filters
// for our exception pivot collections
var summaryResults = from log in messages
group log by log.Exceptiontype into l
orderby l.Count() descending, l.Key
select new
{
ExceptionType = l.Key,
ExceptionCount = l.Count()
};
foreach (var resultItem in summaryResults)
{
// Generate pivots for each exception type
EventLogCollectionSource source =
new EventLogCollectionSource(messages,
resultItem.ExceptionType,
inputFile);
...
targetFilter1.Write(sourceFilter1);
}
Once we we have this code and everything has been generated, if we open the output
folder, we’ll see the following structure:
We see our ExceptionSummary pivot collection and all of the deep zoom folders.
So, when we open the Pivot tool, we’ll see a nice parent collection:
This gives us a nice breakdown of the number of occurrences for each exception in
our source data. Immediately we see an outlier (more on that later) between
the 6,000 and 7,000 item mark and when we select that tile, we see the following:
We also see a green “Open” box (surrounded in a red rectangle, my emphasis) which
links to our NullReferenceException.cxml. When we click that box, the tool will
immediately open that collection in the UI for our perusal – providing a very similar
look to what we saw in the last entry:
Closing Thoughts
Now, you may have noticed a contradiction above. I said that a collection should
have no more than 3,000 items and yet, with the NullReferenceException collection,
we saw in the summary that it had over 6,000 items. That is a very good point
and will be a subject of a future blog post. I wanted to illustrate the simple
collections and the linked collections before we got into that third type of collection
from above – the Dynamic Collection. Stay tuned!