Intro to Filtering with Network Monitor 3.0
Challenges of Filtering
One of the biggest changes between NM2.x and NM3.0 is the way you do filtering. Old NM2.x hacks may be challenged by the loss of the UI wizard to build filters. On the other hand, Ethereal users may be pleased and further encouraged by the built in intellisense, but more on this later. Hopefully I'll be able to ease the transition and provide some tips and tricks for filtering with NM3.0.
What exactly does filtering do?
When you type in a filter and hit apply, each frame is evaluated against the filter. If the result of the filter is TRUE, then the frame is displayed (or captured in the case of a capture filter). So to start simple, if you want to see every frame where the TCP source port is 80, you could type:
Tcp.SrcPort == 80
In this example, the parser engine looks at each frame and evaluates this expression. If the current frames port is set to 80, then it includes that frame in your view.
Where do I start?
Intellisense is technology in many MS products that shows you a list of possibilities given some starting place. In the context of filtering, it allows you to see the available data items for a given protocol or structure. So if you type "TCP." in any of the filter windows you will see a list of available data fields. If you further type in "TCP.Flags." you then see a list of the names for each of the TCP flags. Now unfortunately in the Beta2 version of NM3.0 we only show you two levels deep. So when you type "TCP.Flags." you won't actually see anything more. But in the released version of NM3.0, we should have this implemented fully.
So one little known trick is that you can start off with a "." (period) and you will see intellisense for all the top level items.
In this list you can see stuff like Protocol, which will show you a list of all protocols underneath. So when you are not sure where to start, this is as good a place as any.
Built-in Filters (Standard Filters)
We've actually included a bunch of built in filters with the product. These are common filters that do mostly general things, like target a specific protocol, or narrow traffic down to a specific machine. Once you choose a filter the filter text box is populated with the filter, and you can then apply it. You may have to change some part of the filter, as often they reference placeholder for IP Addresses or Ports. These filters provide a bunch of examples that may help you understand how filters work.
You can also save your own filters and bring them up in other sessions. This lets you access your most used filters easily.
Applying a filter
In order to apply a filter in NM3.0, you can either press the button (paper with a pin on it), or you can hit Ctrl+Enter. One advantage to using the key stroke, Ctrl+Enter, is that it always applies the current filter. When you use the button, you may have to turn off the current filter if you have one applied already. This UI glitch will probably be something we address in the future. Note that we use Ctrl+Enter because our filters can be multiple lines. Having multiple lines helps readability and allows you to add comments as well.
What do I want to filter on?
I suppose it depends on what you are looking for. In general you have some problem you are investigating, or perhaps you simply want to get rid of all uninteresting traffic. Filters allow you to narrow down the traffic and see only the data you want to focus on. You'll often start with something and then further narrow down what you are looking for by adding expressions separated by ORs and ANDs.
Filters can reference anything in the NPL. So this includes protocol data fields like we mentioned above. But this also includes Properties, which are derived by NPL and usually based on the data.
For example, the value of the TCP Window size is a combination of the Windows Scale and the TCP Window Size. So we could create a property to hold the real window size. Each column, with a few exceptions, is also just property value. So this means you can search any column for data in a filter as well. We'll show some examples of this.
Show me all frames where X exists…
So let's start with a simple filter to find all the ARP packets.
ARP
OK, almost too easy. A simple principle to remember is that by simply typing a protocol, structure or property you filter for the existence. So when you type ARP, you are looking for any frame which parses as an ARP protocol. Similarly if you wanted to get rid of all ARP frames, you just say
NOT ARP
Also if you like C like terminology you can type
!ARP
So let's apply this same principle to a structure. Say we want to find all TCP traffic where SACKs are used (Selective Acknowledgments). You don't care what the values of the SACK are, you just want to show only those frames where they exist. So your filter would be:
Tcp.TCPOptions.Option.SACK
And finally, let's apply this to a property. In TCP we have a property called "TCPRetransmit" that gets set whenever a retransmit is found. By the way, this property requires that conversations are enabled, which isn't the default in NM3. So to find all retransmit frames, just type:
Property.TCPRetransmit
Actually the Property portion is only needed if the term was defined as something other than a property as well. Preceding it with the Property suffix makes sure you are referencing it correctly in case it is defined as a structure or protocol too. It never hurts to begin you terms with Property, Struct, or Protocol as called for.
Using equivalence and comparative operators
Another type of filtering that is often required is to look for a specific value in a trace or a value within a certain range. For example, say you wanted to look for traffic on a certain port. In this case you would type a filter like:
Tcp.Port == 5555
This would return any frame with a port (either source or destination) is 5555. What's interesting here is that Port is defined as a "pair". This simply means that we pair up the source and destination port so you don't have to. If you were to explicitly type this out using source and destination ports you would have to type:
Tcp.SrcPort == 5555 OR Tcp.DstPort == 5555
But the Port "Pair" property takes care of this for you. This makes creating filters with paired properties much easier. Since NPL defines these pairs, this could be established for any pair of terms that act this way. Pairs have also been created for Ethernet and IP addresses. So to filter on frames that involve at least one IP address containing 192.168.1.1, you would type:
IPv4.Address == 192.168.1.1
And similarly for Ethernet addresses:
Ethernet.Address==0x1185AE4E95
It's also important to note that when you use this in the negative case, != (not equals), the expansion is different.
Tcp.Port != 5555
Expands to:
Tcp.SrcPort != 5555 AND Tcp.DstPort != 5555
This should be no surprise to you Boolean math heads, but for the commoner this may seem incorrect at first. The typical reaction is to use OR instead of AND. But this will show you only frames where either port is not 5555.
You can also use all the comparative operators like >, <, >=, <=. So if you wanted to search for instances where the window size started getting small, you could so something like.
Tcp.Window < 1200
Now as I mentioned before, if windows scaling is enabled, this may not be the real size. So a better way to do this in NM3.0 is to use the property that calculates the real window size. So this filter would be:
Property.WindowSize < 100
Using Contains to search strings
A common task is to be able to search ASCII and Unicode strings. You can use the Contains plug-in to do this. It searches the associated string and ignores case. This can be used in two different ways, with the same results. Use which ever method you feel more comfortable with.
Contains(property.Description, "error")
You can also use as an operator on a string object.
Property.Description.Contains("error")
As I mentioned before, you can search any property. And remember that most columns are just properties. The exceptions are FrameNumber, TimeOffset, TimeDelta, TimeOfDay, and ConvID. But the rest are fair game. So in this case we can search the description property text using the Contains plug in. Note that using this to search binary data doesn't work. This is something that will probably be addressed in future versions, as it is useful to search binary data.
Generating compound statements by using AND's and OR's:
When narrowing down frames, you'll often start by using one expression to filter the frames down and then you'll want to add other expressions to further restrict your search. So let's say I wanted to look for all frames on port 5555 that also have a window size less than 100 bytes.
Tcp.Port == 5555 AND Property.WindowSize < 100
You can also use the C like shorthand:
Tcp.Port == 5555 && Property.WindowSize < 100
Or to search for traffic between two machines, you could type
IPv4.Address == 192.168.1.1 && IPv4.Address == 10.0.0.1
You have to be careful when using AND's and OR's. The English language tends be more ambiguous when using these terms. Asking a car dealer to see all the Red and Green cars may make sense to both you and him, but the same query to NM3.0 would result in a potentially ugly car that maybe only Santa would appreciate.
Here's another example for filtering on subnets.
(IPv4.Address >= 10.53.0.0) && (IPv4.Address <= 10.53.255.255)
In this example, it's important to understand that the address is really just 32 bits of data. So checking if an address is between the lower and upper bounds of the networks range, is really the same thing as checking if it's in the same subnet.
Using math operators in filters
It's also possible to use math operators, like +, -,*, /,&, and | in filters. The last two being "bitwise and" (&) and "bitwise or" (|). As an example, we can use this to filter out on a subnet, but this time using the "bitwise and". This simulates what subnet mask does.
((ipv4.SourceAddress & 255.255.0.0) == 10.53.0.0)
||
((ipv4.DestinationAddress & 255.255.0.0) == 10.53.0.0)
Comments in the filter window
It's also possible to add comments in the filter window. This helpful if you want to document how a filter works. This also allows you to comment out a section temporarily so you don't have to remove that portion of the filter completely. Comments can be used with either // for a single line comment and /* */ if you want to comment more than one line.
Extra Credit, modifying NPL to filter on a new property IPTTL
We talked before about searching on properties. There may be special cases where you want to create a property yourself so that you can search on it. For this example, we'll create an IPTTL property that can reference both the IPv4 and IPv6 hop count values.
So the first step is to modify both IPv4.NPL and IPv6.NPL to add our property. For IPv4, we'll add the property to the TimeToLive data field and for IPv6 the HopLimit data field. In NPL properties are placed in square brackets "[ ]" before the data field definition. The bracket section can contain multiple lines separated by commas but we won't have to worry about this for our example.
In IPv4.NPL, we see that the TTL parameter is defined as follows.
…
};
UINT8 TimeToLive;
[NextProtocol]
UINT8 NextProtocol = FormatString("%s, %d(%#x)",
ProtocolTypeTable(this), this, this);
So what we do is add in property called IPTTL as follows:
[IPTTL]
UINT8 TimeToLive ;
The property automatically attaches itself to the data that follows it. So now let's modify the IPv6 parser. Here's how it looks in its original form.
[NextHeader]
UINT8 NextProtocol = FormatString("%s, %d(%#x)",
ProtocolTypeTable(this), this, this);
UINT8 HopLimit;
So we'll make the same change here:
[IPTTL]
UINT8 HopLimit;
The properties are named the exact same, so we now can reference the same property name. If IPv4 exists it references TimeToLive, and if IPv6 exists we reference HopLimit.
You must now reload the parsers. This is similar to a recompile as we want to save time each time you run NM3.0. This can be done from the parsers tab. You can hit the button from the tool bar, select Tools Reload Parsers, or type Ctrl+Alt+B.
Once the parsers are reloaded, you can then use this in a filter as follows:
Property.IPTTL == 0
This will return all frames where the TimeToLive or HopLimit is set to zero.
And that's your filtering introduction
While it takes a while to get good at filtering the discussion above should give you a good premier. Hopefully this will help you understand the basics of how to use filtering to find the data you want with Network Monitor 3.0.
C:Documents and SettingspaulloMy DocumentsNetmon projectBlogIntelliExample.bmp
Comments
Anonymous
January 01, 2003
One of the limitations of contains is that it only works with Ascii or Unicode data. And in this case, Telnetdata is defined as a struct, and the data with in described as a bunch of UINT8's. Hopefully we can address this in the future. But NPL to the rescue. The Telnet data isn't always Ascii, so we can't define it as such. But we could modify the NPL so that properties are availble to search on for both Ascii and Unicode. So I've changed Telnet.NPL as follows. struct TelnetData { /while cond[ (offset < FrameLength) && (FrameData[offset] != 0xff)] { UINT8 Data; }/ [TelnetAsciiData = AsciiString(FrameData, FrameOffset, FrameLength - FrameOffset), TelnetUnicodeData = UnicodeString(FrameData, FrameOffset, FrameLength - FrameOffset)] ArrayAsHex(FrameLength - FrameOffset) Data; }; Once you make this change and reload the parsers, you can now search on the property as follows: property.TelnetAsciiData.contains("login") This should find the frames you are interested in. Thanks, PaulAnonymous
January 01, 2003
To create a filter by time of day is not as easy as it should be. Right now we have an internal reprentation of time as a 64 bit number which is an offset from the start time, but you can't enter stuff like 12:45 in a filter. The way to specify a time is to click on the frame in question and right click the Time Offset column and select "Add 'Time Offset' to filter. You'll see the 64bit value for that frame and understand how to convert using an offset. You can also use this to find the range by selecting the first and last frame, then changing the filter to use <= and >= rather than ==. BTW, NM3.2 is the latest released version and NM3.3 beta was just released on codeplex. Thanks, PaulAnonymous
January 01, 2003
If you are asking if it's easy for 3rd parties to create parsers, then yes. Creating parsers is fairly straight forward and you don't need anything more than what is included in the download. Learning the NPL langauge is the main struggle, but far easier than creating a DLL for Netmon2.x. (See my blog on the parsers for more info). NM3 will continue to be a product that evolves over time. We have already started work on 3.1 and should have a Beta available, hopefully towards the end of Summer. Thanks, PaulAnonymous
January 01, 2003
The comment has been removedAnonymous
January 01, 2003
The comment has been removedAnonymous
January 01, 2003
Thanks! Uncommenting the definition of TCPPayload in the TCP parser worked. I was able to perform the search I need using this property. Is this property planned to be uncommented and available in the released product? Also, are there any plans to add an expression filter "wizard" or GUI to help the transistion to creating filters? Thanks again, JohnAnonymous
January 01, 2003
OK, I'm feeling pretty stupid here... I've been messing with this program (for my first time - Netmon 3.1.512.0) and I figured it might come in handy to create a Display Filter to show the items captured between this time of day and that one. Even right-clicking and "Add Cell to Display Filter" doesn't work - every time I create an expression and Verify it, it returns a blank list of captured packets. I'm sure that once I use this more I will dream up more meaningful filters, but right now I am trying to track down an intermittent communications problem that happens once a week or so (that I know of) and the time of which I can pinpoint using backup software logs and the Event Log. Wouldn't it make sense, once I've captured several weeks' worth of packets, to be able to display only those packets that appeared during a certain time span? The documentation and blogs I've found so far don't mention the use of a time filter - maybe there's another way to do it? Thanks,Anonymous
January 01, 2003
Paul, thanks a lot - I'll give it a bash on Monday when I'm back at work. The ability to create new structs seems pretty powerful. Is it fairly easy for third parties to use this to create extension to NM? Also, when you talk about 'addressing in future' is NM3 going to be a product that evolves over time, rather than stagnates like NM2 seemed to (and apologies if this wasn't the case) JamesAnonymous
January 01, 2003
This is good stuff, but where has all the network statistic information gone?Anonymous
January 01, 2003
This probably won't be enabled in the released build. We wanted to stabilize the NPLs, so we are not making any changes to the NPL till after release. Hopefully we'll have a way to update the parsers separately from the product. As for an expression filter, we have put thought into providing something to bridge the gap. But we are still evaluating it's usefulness. Thanks, PaulAnonymous
January 01, 2003
Excellent post. I would have been lost without this. I couldn't get either of the contains methods to work searching TCPPayloadData. I was able to do this type of serach with the old version of Netmon. Is this an issue because TCPPayloadData is not listed under .Property but under .Protocol.TCP.TCPPayloadData? The contains I'm trying to use is: and .Protocol.TCP.TCPPayloadData.Contains("$SOM") Looking for packets that contain the string "$SOM". JohnAnonymous
January 01, 2003
thanksAnonymous
January 01, 2003
The comment has been removedAnonymous
January 01, 2003
If you are refering to the statistics included with Netmon2.x, we have not added that back in yet. With the API for NM3.2, it should be possible to create some of these items based on the trace data. We will be considering the re-addition of this type of information for future versions of NM3. Thanks, PaulAnonymous
January 01, 2003
Great article, and echo the 'would have been lost without this' comment. Having trouble however getting the 'contains' property to work. Have set a display filter as follows : protocol.telnet.option.Telnetdata.contains("Login") Within the capture file I do have data matching this, ie : 4C 6F 67 69 6E 20 53 75 63 63 65 73 73 66 75 6C 22 20 20 20 20 Login Successful but I get no packets returned. Doing a simple filter, ie 'telnet' works fine. Am on version 3.0.372.1 running on XP SP2. Any ideas what I might be doing wrong?Anonymous
January 01, 2003
thanksAnonymous
January 01, 2003
thanksAnonymous
January 01, 2003
You just need to change the AND to OR, and you'll see frames where either process name matches. The AND tells it to look for a frame where the process name is javaw.exe AND sslump_nxd.exe, which can never happen. When you apply any filter, put it in the context of a single frame and perhaps that will help you understand how to build a filter to find the data you need. Also wanted to make sure you are on our latest version, 3.4, as 3.2 is pretty old now. PaulAnonymous
October 25, 2010
The comment has been removedAnonymous
February 17, 2012
The comment has been removedAnonymous
February 20, 2012
Thanks for your reply. But I want the results which donot contain ".51.168.... is there a way to match .!contain(".51.168...) ?Anonymous
December 18, 2012
Is searching the description property text using the Contains plug in the only kind of ASCII searching that can be done? I was hoping to search an entire (albeit short) capture file for an ASCII string. I used a generic hex editor to do an ASCII search for a string, which it happily found, but of course there is no way to know in which frame the string is contained.....thanks in advance!Anonymous
April 10, 2014
Great Article..Anonymous
November 15, 2015
The comment has been removed