通过

Making things go fast on a network, part 2

  • 项目

Sorry about the delay - I got nailed by a nasty sinus infection on Tuesday night last week that took me out until today.

 

In my last post I started discussing some of the aspects of networking that need to be understood before you can make things "go fast" on a network.

As a quick recap, here are the definitions and axioms from that post (note: I've changed the definitions of packet and message because they make the subsequent articles easier):

First, some definitions:

  1. When you're transferring data on a connection oriented protocol, there are two principals involved, the sender and receiver (I'm not going to use the words "client" and "server" because they imply a set of semantics associated with a higher level protocol.
  2. A "Frame" is a unit of data sent on the wire. 
  3. A "Packet" is comprised of one or more frames of data depending on the size of the data being sent, it typically corresponds to a send() call on the sender
  4. A "Message" is comprised of one or more packets of data and typically corresponds to a higher level protocol verb or response.

And some axioms:

  1. Networks are unreliable.  The connection oriented protocols attempt to represent this unreliable network as a reliable communication channel, but there are some "interesting" semantics that arise from this.
  2. LAN Networks (Token Ring, Ethernet, etc) all transmit messages in "frames", on Ethernet the maximum frame size (or MSS, Maximum Segment Size) is 1500 bytes.  Some of that frame is used to hold protocol overhead (around 50 bytes or so), so in general, you've got about 1400 bytes of user payload for in each frame available (the numbers vary depending on the protocol used and the networking options used, but 1400 is a reasonable number).
  3. A connection oriented protocol provides certain guarantees:
    1. Reliable delivery - A sender can guarantee one of two things occurred when sending data - the receiver received the data being sent or an error has occurred.
    2. Data ordering - If the sender sends three packets in the order A-B-C, the
      receiver needs to receive the packets in the order A-B-C.

At the end of the last post, I introduced one consequence of these axioms:  When sending packets A, B, and C, the sender cant transmit packet B until the receiver has acknowledged receipt of packet A.  This was because of axiom 3.b.

There's one thing I forgot in my last post:

What happens when the receiver isn't ready to receive data from the client?

Well, it's not very pretty, and the answer depends on the semantics of the protocol, but in general, if the receiver doesn't have room for the packet, it sends a "NACK" to the client (NACK stands for Negative ACKnowledgement).  A NACK tells the client that there's no storage for the request, the client now needs to decide what to do.  Sometimes the NACK contains a hint as to the reason for the failure, for instance the NetBEUI protocol's NACK response includes the reasons like "no remote resources, unexpected request, out of sequence".  The client can use this information to determine if it should hang up the connection or retry (for no remote resources, for instance, it should retry).

Sender Receiver
Send Packet A.1  
  Send ACK A.1
Send Packet A.2  
  Send NACK A.2 (No Memory)
Send Packet A.2  
  Send NACK A.2 (No Memory)
Send Packet A.2  
  Send ACK A.2
Send Packet A.3  
  Send ACK A.3

All of this retransmission goes on below the covers, applications don't typically need to know about it.  But there's a potential perf pitfall here.

 

If you're analyzing network traces, you often see this pattern:

Sender Receiver
Send Packet A.1  
  Send NACK A.1 (No Memory)
Send Packet A.1  
  Send NACK A.1 (No Memory)
Send Packet A.1  
  Send NACK A.1 (No Memory)
Send Packet A.2  
  Send ACK A.2
Send Packet A.3  
  Send ACK A.3

What happened here?  Well, most likely the problem was the receiver didn't have a receive buffer down waiting for the sender to send data, so the sender had to retransmit its data to the receiver before the receiver got around to being able to receive it.

So here's "Making things go fast on a network" perf rule number 1:

        Always make sure that you have a receive request down BEFORE someone tries to send you data.

In traditional client/server networking, this rule applies to clients as well as servers - a client that doesn't have a receive outstanding when the server sends the response to a request, it will stall in the same way waiting on the client to get its receive down.

 

Btw, a piece of silly trivia.  J Allard, of XBox fame used to have two iguanas in his office named ACK and NACK back when he was the PM for NT networking.

Comments

  • Anonymous
    May 15, 2006
    I think you meant "What happens when the receiver isn't ready to receive data from the sender?"

    You weant and used the 'c' word :-)

  • Anonymous
    May 15, 2006
    > If you're analyzing network traces, you often see this pattern

    I'm confused.

    Due to windowing I wouldn't be surprised to see sends of A.1, A.2, and A.3 before the first NACK for A.1.  Depending on the protocol and the recipient's buffering policy I wouldn't be too surprised to see ACKs for A.2 and A.3 while waiting (prepared) for A.1 to be retransmitted.

    But if the sender repeats A.1 due to NACKs, then why did the sender ever proceed to A.2 in the first place?  If the sender is retransmitting A.1 until it gets an ACK then how does your pattern ever get seen?

  • Anonymous
    May 16, 2006
    The comment has been removed

  • Anonymous
    May 17, 2006
    maybe I'm being obtuse, but shouldnt the last Send Request of A.1 (of 3) have gotten an ACK?

  • Anonymous
    May 18, 2006
    Here again, Larry, I think your presentation may be NetBEUI-centric (or some other protocol that I'm not familiar with). TCP handshaking establishes a receive window size; the sender can't send until it knows there's a nonzero receive window. Stevens has a basic discussion of this in Chapter 20 of Volume 1 of TCP/IP Illustrated and spends some time talking about bandwidth-delay product and determining the optimum window size.

  • Anonymous
    May 24, 2006
    I don't think there are many developers today that actually get this. They are using abstractions that hide all this. Really nice to get it mentioned.

  • Anonymous
    June 07, 2006
    Reply to nick about recv:

    Yes, it's not necessary to get the receive down before the data arrives at the node. But the focus of these articles is speed, and if you're interested in speed, you want to have that receive in place. Why? Because the OS will go out of it's way to land the data into the buffer you provide as it's received. It does this for the obvious efficiency reasons; if it doesn't, a data copy is required to get the data from the transport buffer to the application buffer. And that copy is relatively expensive.
    Note that this apparently even more true in Vista/Longhorn, from my (admittedly brief) looks at it so far.

  • Anonymous
    June 12, 2006
    The comment has been removed

  • Anonymous
    June 13, 2006
    The comment has been removed

  • Anonymous
    June 26, 2006
    Oh, Larry, Larry, Larry...
    Articles 1 and 2 were great - really necessary reading to a lot of would-be...

  • Anonymous
    June 09, 2009
    PingBack from http://weakbladder.info/story.php?id=5204

  • Anonymous
    June 18, 2009
    PingBack from http://thestoragebench.info/story.php?id=62