次の方法で共有

Reassembly with NM3

  • [アーティクル]

Ever wonder how a network works? Maybe it just seemed so easy, and in your mind sending a file was just putting each byte on the wire to the receiving machine. That’s not too far from the truth but you’d be very selfish to think that the network was there for your bidding only. Since a network has to be shared between many users, strategies have been created to chop your file up and send it in chunks.

Like Packing for a Move

We’ve all done it before. We take all of the bits and pieces of junk we’ve collected over the years and are tasked with moving them from one location to another. We invite friends and relatives, and like ants, they arduously move each piece from house A to house B. If your house=file, and boxes=packets, then we’ve created an analogy for reassembly. If you are organized, you’ll label each box, 1 of 20, 2 of 20, and so on. When you get to the new location, you can put all your things back together using the labels you created and verify you received everything.

The architects of our Internet are not any cleverer. They’ve borrowed these same techniques to determine how a chopped up file can reach its destination and be reconstituted. As long as the data is sequenced, the other side should be able to put the pieces back together.

The Transport Layer

In networking we like to talk about layers. You may have heard terms like “Layer 3 switch” or the “Network Layer”. Well, the architects of our Internet are just like everybody else. They get easily confused if too much is going on. Transferring network data from one place to another is a difficult problem. And a common strategy used to solve difficult problems is to divide and concur. Each layer is responsible for different things, and the transport layer is the one we use to define how files get chopped up and rebuilt.

We’ll narrow our focus down to TCP for a bit. This is the workhorse transporter for most networking applications. There are others, such as SPX/IPX, but TCP is by far the most popular. And it turns out RCP, SMB and HTTP can also fragment data. SMB might chop your file up into 4K chunks, and then each of these chunks could be further fragmented by TCP.

Labeling Your Boxes

So in our analogy, labeling=sequencing. We could label each packet 1 of 20 and so on, and some protocols do use this strategy. But for TCP, we go a bit further and describe the number of bytes that are sent. We’ll say, for instance, that this packet contains Sequence 1000-2000. What we actually send is the first sequence number and the size, but this range can be derived from that data. When the other side gets the data, even if it’s out of order, it knows how to put the puzzle back together. Also, the receiving TCP will keep the sender up to date by acknowledging what it’s received so far. In the simplest scenario, the receiver ACKs the latest segment it received. If the sender gets an acknowledgment for sequence 2000, then that confirms the receiver has seen all data up to that point.

Using NM3 to Reassemble the Data

When we capture data from the network, we are capturing it before the data has been put back together. But this can make it difficult to read, as you might imagine. With NM3 we’ve created a way to reassemble the data, so that you can see the data as it is seen by the application layer (there’s one of those darn layers again, see https://en.wikipedia.org/wiki/OSI\_model). So when you click on a web page and get a trace, you can see the entire packet as sent by the browser, rather than a bunch of fragments after TCP has gotten to them.

Data Before You Reassemble

So if you start a network trace and view a web page you’ll notice that the traffic that gets created shows HTTP and TCP. The HTTP packets are the headers sent/received by your web browser. But the TCP that is in between is also your browser traffic. The original HTTP packet was larger than would fit in single packet, so TCP has chopped it up for you.

In this example, you can see that the server has responded with a page. Frame 6 is a continuation of frame 5 and is where TCP has chunked up the data. Frame 7 is an acknowledgment so the server knows we are receiving data. And frame 8 is the final frame in the fragmented data.

Frame Source Destination Description
5 Srv Client HTTP:Response, HTTP/1.1, Status Code = 502, URL: https://239.255.255.250/
6 Srv Client TCP:[Continuation to #5]Flags=...A...., SrcPort=HTTP(80), DstPort=49382, PayloadLen=1460, Seq=3331697971 - 3331699431, Ack=1190309335, Win=65217 (scale factor 0) = 65217
7 Client Srv TCP:Flags=...A...., SrcPort=49382, DstPort=HTTP(80), PayloadLen=0, Seq=1190309335, Ack=3331699431, Win=255 (scale factor 8) = 65280
8 Srv Client TCP:[Continuation to #5]Flags=...AP..., SrcPort=HTTP(80), DstPort=49382, PayloadLen=1392, Seq=3331699431 - 3331700823, Ack=1190309335, Win=65217 (scale factor 0) = 65217
Data After You reassemble

To reassemble with NM3.1, you go to the Frames menu and select “Reassemble All Frames”. In NM3.2 we’ve created a more prominent button on the tool bar so this should be easier to find. Once the reassembly is complete a new window opens and contains all the original frames PLUS new frames for each reassembled piece.

Frames Source Destination Description
5 Srv Client HTTP:Response, HTTP/1.1, Status Code = 502, URL: https://239.255.255.250/
6 Srv Client TCP:[Continuation to #5]Flags=...A...., SrcPort=HTTP(80), DstPort=49382, PayloadLen=1460, Seq=3331697971 - 3331699431, Ack=1190309335, Win=65217 (scale factor 0) = 65217
7 Client Srv TCP:Flags=...A...., SrcPort=49382, DstPort=HTTP(80), PayloadLen=0, Seq=1190309335, Ack=3331699431, Win=255 (scale factor 8) = 65280
8 Srv Client TCP:[Continuation to #5]Flags=...AP..., SrcPort=HTTP(80), DstPort=49382, PayloadLen=1392, Seq=3331699431 - 3331700823, Ack=1190309335, Win=65217 (scale factor 0) = 65217
9 Srv Client HTTP:Response, HTTP/1.1, Status Code = 502, URL: https://239.255.255.250/

So frame 9 is the new frame we inserted. While the Description doesn’t look much different, the devil’s in the details. Quite literally we have to look at the frame details to see the difference.

 Frame: Number: 9, Captured Frame Length = 4449, MediaType= PayloadHeader<br>+ PayloadHeader: Re-assembled Payload<br>+ Http: Response, HTTP/1.1, Status Code = 502, URL: https://239.255.255.250/

The major difference is that we don’t see Ethernet, IP, or TCP anymore. We’ve replaced the original network header with our own called PayloadHeader. This header contains info about the protocols that have been reassembled, as well as information we might need from those layers. We don’t show the original frame numbers, but you can enable some debug NPL to get this information if you need. Just look at Payload.NPL for more info.

If you were to open up HTTP Response in the details, you would also see that all the HTML and header information is in this one packet. This reassembling of the data makes is very easy to understand that data from the HTTP level.

Also realize that in cases where both HTTP and TCP are fragmenting data, there can be multiple levels of reassembly. In those cases you may see a PayloadHeader for each TCP fragmentation, and then another for the HTTP fragmentation.

Reassembly FAQ

Reassembly doesn’t seem to be working for me, why?

The reassembly feature does depend on two things. One, conversations have to be enabled. Make sure this option is enabled when you open a trace. Currently the option can be found on the start page. Two, you can only reassemble a saved trace. If you just started a trace, you’ll need to save and reopen it.

Why do all the original frames show up?

When NM3 does reassembly, new frames are added in. Rather than remove the original frames, we’ve decided to leave them in as they still provide information about the original TCP data. Also, if there’s a problem with reassembly, perhaps due to missing packets, you could use the data to figure out why it didn’t reassemble correctly.

Can I only see reassembled frames?

Well yes, you can filter on only those things that have been reassembled. As we mentioned above, we add a PayloadHeader, which is just a protocol we’ve devised. So you can filter out all reassembled data, but applying a filter of “PayloadHeader”. I actually create a color filter for this so they stick out.

What you can’t do, however, is see only data that has been reassembled and the data that was never originally fragmented. While we realized this would be useful, we haven’t exposed a way to make this possible. However, with the API and NM32, it would be possible to do this programmatically.

Why doesn’t the Reassembly window have a conversation tree?

When we originally designed reassembly, the conversation window wasn’t really integral to the product yet. The simple work around is to save the capture file and reopen.

Putting it All Together

Networks have devised ways to break apart data and rebuild them. But with protocol analyzers that capture this raw data, it’s sometimes difficult to follow. With the reassembly option in NM3 your network traffic is put back together making it easier to read. Add this ability with the new Process Tracking feature in the upcoming NM3.2 and finding the only the data you need will be easier than ever.

Comments