Clicky

How to Troubleshoot Throughput and TCP Windows

By Kary | tutorial

Jul 30

Make sure you’ve read Understanding Throughput and TCP Windows before watching this video. I mean, you don’t HAVE to, but I recommend it.

No one’s ever asked you why the network is slow, right? Hahahahahaaaaaaa haa ha. Ha. Oh man. What a funny joke. The better question is can you think of a day in which someone DIDN’T ask you that question?

It’s frustrating because that one time 6 months ago when it was the network means that since then anytime someone’s cat video doesn’t instantly load they’re asking you about the network. And it seems like you have no way to tell them to get lost because how can you prove it’s not the network?

I’m here to tell you that you can prove it and I gave you one example of when it’s not the network in the last video. Now I’m gonna show you a couple more.

In this video, I walk you through two captures. One shows throughput bound by the receiver and the other by the sender. Again, if terms like “receive window” or “congestion window” or “send buffer” don’t make sense to you, go read the link above first.

I sometimes worry about how long these videos are, but the thing is I’m only covering maybe 75% of what I could talk about in the capture files. There’s just so much information and I don’t want it to be overwhelming. I’m trying to give you a depth that you can’t get much place else. If you really want to learn this stuff, you’ve got to be willing to make an investment of your time. Just watching a 5min video ain’t gonna do it. Feel free to leave a comment about this topic.

Your Turn Challenge

Grab the sendwin-bound.pcapng file and tell me in the comments why sometimes it sends only 64k before waiting for an ACK when most times it sends 128k.

Also, if you’d like to look it over, here’s the recvwin-bound.pcapng

Attribution:

Entry of the Gladiators

Share this post! Spread the packet gospel!

Facebooktwittergoogle_plusredditlinkedinmail
Follow

About the Author

I like being the hero. Being able to drop a bucket of root cause analysis on a burning network problem has made me a hero (to some people) and it feels real good, y’all. Get good at packet analysis and be the hero too. I also like french fries.

Leave a Comment:

(10) comments

Michael July 30, 2014

So, I’m looking through the capture file, and I’m seeing a two byte packet go out after it receives ACKs for the first 64k. Most of the time it’s including those two bytes with the final 128k ACK packet. In the cases where it’s only sending 64k at a time, that two byes has not been included in the final ACK. As for why the server isn’t ACKing those two byte packets in a timely manner, I’m really not sure.

Reply
    Kary July 30, 2014

    Nice, Michael. That is indeed the pattern that triggers it. Clearly having that 2 byte segment unACKed causes the sender to stop after 64k+2 bytes and wait for a single ACK. Then it sends the second 64k chunk. It’s not clear to me why it pauses when it can send more than 64k bytes in the other times it sends 128k. But something about that 2 byte segment being unACKed causes it to pause. Good eye!

    Reply

[…] How to Troubleshoot Throughput and TCP Windows […]

Reply
chrismarget July 31, 2014

What’s the sending stack?

The PSH bit being set on the end of that 2-byte segment suggests to me that it falls at the end of a socket send() operation.

My suspicion is that there’s a zero-copy mechanism working inside the sending stack: On paper, ACKed data should be purged from the sender’s send buffer segment-by-segment as ACKs roll in, guaranteeing that new data from the sending application is always available for the stack to send.

Zero-copy mechanisms don’t copy data into the socket buffer. Instead, they say “hey, stack! here’s a pointer to a block of data I’d like you to send!” These mechanisms are nice in that they avoid the copy operation (speedy!), but they quickly expose tuning problems because the stack winds up working with very large chunks of data as atomic units. The stack can’t clear individual segments from its “buffer”, because it’s trading pointers to very large chunks of memory. The block of memory and the pointer are both busy until the last ACK for that chunk is received.

If there’s enough memory, and enough pointers, then we’ve effectively got a windowing mechanism (inside the stack) exactly like TCP’s byte-based windowing. You’ll never know that pointers to these large data chunks are cycling around inside the sender.

If these resources inside the sender are scarce, then you run into ugly business like this.

Sometimes the application is complicit in this scheme (google: “io completion ports” for an MS Win example), and sometimes not.

Reply
    Kary July 31, 2014

    Very good info, thanks, Chris. The sending stack is Win7. Don’t know the exact version; hafta check. iperf is calling write() with 128k of data to a 64k send buffer. In my research, I’ve seen that the kernel will fudge a bit on the actual buffer size so I assumed this was why it would put more on the wire than the send buffer size. But perhaps the zero-copy behavior is why it will put 128k on the wire (except when the 2 byte segment is unACKed) when there’s only a 64k buffer. I’ll research io completion ports. Thanks for the tip!

    Reply
David Zhang December 31, 2014

Hi Kary,
Great post!!! Very helpful. Could you please share the second packet capture which you refer in the video?

Best Regards,

Reply
Lleyton October 23, 2015

Dear Kary,

Thanks. This is very helpful. Could you pls share the 2nd pcap? The Rx one.

Reply
    Kary October 25, 2015

    Sure, added to the post

    Reply
Brad May 24, 2016

Michael/Kary,

For the purpose of learning, could you provide a step by step explanation on how you came to that conclusion?

Reply
Add Your Reply

Leave a Comment: