Clicky

How to Prove It's Not the Network

By Kary | tutorial

Jul 07

Or prove that it is the network.

If you’re a network engineer, how much time do you spend defending the network? Everyone points a finger at the network; it’s guilty until proven innocent. Everything looks good from your standpoint: monitoring, alerts, maybe some pings, download a file from a server…it all seems to check out. So you point fingers back. But how do you definitively exonerate the network?

Maybe you’re a sysadmin your application server is running in the green. CPU, memory, interface stats all check out. So you point a finger at the network. But how can you be sure there’s not an issue lurking somewhere with your server?

Maybe you’re an application developer on the client or server side. It’s not your code, right? It’s definitely the network, or maybe IT needs to give you a bigger server because it doesn’t have enough horsepower. But are there some clues that could let you know if your app has a logic or inefficiency issue?

That’s where packet analysis comes in. Through this video and more to come, I’m going to show you how to dig in and start eliminating things and finding the right direction to do your investigation, whether you are a network guy, a server girl, or a developer chipmunk.

Here’s a hypothetical scenario: Ol’ Betty in accounting (who is also the CEO’s mother) is having trouble downloading her cat pictures and she’s real upset about it. If Betty’s upset, we don’t get paid on time. So it’s in everybody’s best interest to get this solved.

It’s a good idea to get data as close to the source of the complaint as possible, to see it from their perspective. So capture on Betty’s machine if possible or span her port or use a hub between her computer and the wall port. Now, I’m going to skip over a lot of important steps before you get to the analysis: asking the right questions, narrowing down the problem, getting the data, etc. We can cover those things in the future but in the interest of this video not being two hours long, let’s get to the analysis.

This is just one possible outcome of the scenario. There are endless possible outcomes and analyses. For example, I just finished troubleshooting a Windows 7 upload performance issue that is completely different than this. I’ll be writing it up soon; I learned more about Windows TCP than I ever thought I would.

Here’s the Wireshark setup videos I mentioned:

Your Turn Challenge

I’ve got two pcaps for you: the FTP one from the video and an iperf one for you to analyze. Analyze the iperf pcap and leave a comment or email me and tell me where the delay is: client, server, network, etc. Hints:

  • What looks weird about the TCP Time-Sequence graph?
  • Add a “delta” column and sort by it
  • Who are we most consistently waiting on? Client or server?

Download here

P.S. The iperf capture is a sneak peek of the next thing I’m working on.

Share this post! Spread the packet gospel!

Facebooktwittergoogle_plusredditlinkedinmail
Follow

About the Author

I like being the hero. Being able to drop a bucket of root cause analysis on a burning network problem has made me a hero (to some people) and it feels real good, y’all. Get good at packet analysis and be the hero too. I also like french fries.

Leave a Comment:

(14) comments

John July 8, 2014

Hi Kary – great post. Reading packet captures is always something I wish I had a better understanding of. From what I can see it looks like the delay is on the client side.

Reply
    Kary July 8, 2014

    Hi John,

    Nice one! Yes, it is on the client side. I’ll be digging into why in a future post, but if you have any ideas, let’s hear ’em!

    Reply
      John July 9, 2014

      Looks like a delay in ACKs from the client to the server.

      Reply
        Kary July 11, 2014

        When I say client, I’m talking about the host that initiated the connection. In this case with iperf, the client is the one sending data. So when I say the delay is on the client side, I mean the side sending data. See Chris’ comment below on the cause of the slowness. You are correct in that the delay before the ACK is the delay, but that delay is caused by the sender’s (client) too small send buffer for the BDP of the connection. I’ll go into more depth in a future post.

        Reply
chrismarget July 10, 2014

In the iperf capture, 10.2.0.1 has a too-small transmit buffer (64KB-ish) to make good use of this long (90ms), fat (ACKs are rolling in at 9MB/s!) path.

Increasing the send window will help until you hit 1.2.3.4’s receive window 500KB-ish. You should be able to get up to around 5MB/s with changes to the send window only.

Ever use tcptrace and xplot? Sooo much better than visualization in wireshark. It’s the best tool for this work in my opinion.

Reply
    Kary July 10, 2014

    Very nice, Chris. It is indeed send buffer limited. Now, can you tell me why Win7 isn’t dynamically increasing it as my research suggests that it should? Because damned if I can figure out why. :-)

    I have used tcptrace and xplot and will definitely cover it in the future.

    Reply
      chrismarget July 12, 2014

      Kary,

      > Now, can you tell me why Win7 isn’t dynamically increasing
      > it as my research suggests that it should? Because damned
      > if I can figure out why.

      10.2.0.1 understands window scaling, but sent a bone-headed scaling factor of zero.

      Now, that’s on the receive side, but maybe there’s something equally stupid going on with the send side that we can’t see on the wire?

      Reply
How to Troubleshoot Throughput and TCP Windows July 30, 2014

[…] tell you that you can prove it and I gave you one example of when it’s not the network in the last video. Now I’m gonna show you a couple […]

Reply
iqbal August 19, 2014

Great explanation.. I just started to learn packet analyze and this is a great video!
Thanks Kary.

Reply
r00t0xic September 12, 2017

Just watch ,Love it!!

Reply
Rob September 26, 2017

I realize this is an old post. When I sort by Delta Time I see the large lags in ACK delays from 1.2.3.4 to 10.2.0.1

I just haven’t been able to decipher exactly what your’re pointing to as the cause and where in Wireshark that can be viewed.

Am I correct is saying 10.2.0.1 is what this discussion is calling “the client” some confusion with that. It looks like data is mostly going from 10.2.0.1 to 1.2.3.4

So I’d say 1.2.3.4 is the server and we’re waiting on the server. But I know that’s not what the answer is by reading above. Can somebody clarify?

Reply
    Rob September 26, 2017

    I think I worked out my own question after digging around…..I guess if anybody wants to still reply I’d happily read it.

    Reply
    Kary September 26, 2017

    Rob, this content is a timeless masterpiece! That said, I can’t answer your question right now but this post was too long ago :) Let me review and get back to you

    Reply
      Rob September 26, 2017

      Oh that’s okay. I finally understood the explanation. Thanks for the very useful posts/videos. Much appreciated. I hadn’t seen your site before.

      Reply
Add Your Reply

Leave a Comment:

Previous Post Understanding the tcptrace Time-Sequence Graph in Wireshark Next Post Why Should You Learn Packet Analysis?