Clicky

Why Should You Learn Packet Analysis?

By Kary | text

Jul 16

lightsaber dick

(I’ll explain this picture later.)

This whole thing started when I posted Why don’t more people know how to do packet analysis? on reddit. Up to that point, I thought maybe it was just me and a few others who valued and dug packet analysis since I hadn’t met many people who could do it really well. That thread showed me that people do understand the value, but that it’s difficult to get started.

So how do you learn? Let’s talk about the first step.

In the past, I’ve found things that interest me and that I want to learn. Like I wanted to learn how to program an iPhone app so I bought a book. I kinda learned some stuff but I never built anything. I used to do some video production and I wanted to learn After Effects so I bought a book. I kinda learned some stuff, but did I ever do anything other than a stupid lightsaber joke with AE? No. I wanted to learn how to breakdance. I watched a bunch of YouTube videos and cleared out my living room floor.  I’ll let you guess how that went.

The point is I didn’t have a burning desire or reason to learn. So I didn’t. I just thought it was cool.

Using packet analysis for finding root cause is a skill. I reckon there’s an art to it as well. You can learn but it takes time and effort. If you don’t have a good reason that will help keep you at it, stop now. You might as well click on to the next thing. Have you seen the gymnast girl completing the Ninja Warrior course? It’s real popular right now. Go enjoy that.

Here are some reasons off the top of my head why you might want to learn packet analysis:

  • Tired of the network being blamed every time there’s a problem and you can’t definitively defend it
  • Tired of your application being blamed for being slow
  • Tired of your server being blamed for being slow
  • Find root cause quickly and shut the nay sayers up
  • Eliminate finger pointing
  • Sometimes it’s the only way to be sure about an issue
  • Stop troubleshooting by trial and error
  • For the knowledge (know yer shit)
  • Don’t want to be a paper tiger (certs but no real knowledge of what’s on the wire)
  • Being the expert
  • Being the best
  • Being the hero
  • Getting the promotion, $$$$, new position, etc

I get a lil high from solving problems, especially hard ones that other people couldn’t solve. I remember meeting an engineering manager from another office for the first time and he said, “So you’re the one my guys talk about that can analyze traces like the wind.” That felt real fuckin’ good, y’all. You can be That Guy too. It just takes consistent effort and a desire to do it.

Leave a comment and tell me:

  • Why you want to learn or get better at packet analysis
  • A time when you saved the day with packet analysis

Identifying why you want to learn is the first step. We’ll talk about the next step in the future.

Share this post! Spread the packet gospel!

Facebooktwittergoogle_plusredditlinkedinmail
Follow

About the Author

I like being the hero. Being able to drop a bucket of root cause analysis on a burning network problem has made me a hero (to some people) and it feels real good, y’all. Get good at packet analysis and be the hero too. I also like french fries.

Leave a Comment:

(4) comments

chrismarget July 18, 2014

My best analysis story:

An application critical to my company exhibited performance problems, was falling on its face in customer deployments. It was a stock pricing application used at the head of ticker plants in financial companies all around the world. If you had a 401(k) around 2000, it probably depended on this application. I did analysis of the sort you’ve been describing, specifically TCP behavior. I pinpointed the problem as being in the OS vendor’s implementation of TCP. The buggy behavior was that whenever the sending stack went into congestion control, it never recovered. This resulted in a comically small send window, sometimes just a few multiples of MSS.

It took a while battling with account managers and developer support folks at the OS vendor who didn’t understand the problem, my explanation, or that the issue *couldn’t* be in the application because the application is blissfully ignorant of TCP machinations. It was like talking to a wall. I started at square one with every conference call. Eventually I got on the phone with a guy with whom I could have a good discussion. It turns out that he put the RFC1323 extensions into the stack! The next day I had a patch to the OS in my hands and the product worked perfectly from that point forward.

The developer explained that there was a bug which caused incoming ACKs *with payloads* to be miscategorized as DUPACKs when the stack was in congestion control.

This would never happen with half-duplex applications like HTTP, but the application I was supporting sent data bidirectionally on the socket at all times.

I didn’t have a ton of support from management at the time (my manager even yelled at me for “always wanting to use a sniffer” to fix problems), and nobody but me was looking at the OS vendor’s TCP implementation as the source of the problem. Wrestling the fix from the OS vendor by myself made this victory particularly sweet, earned me a ton of capital to do my own thing, and led to the most interesting problems showing up on my desk.

FWIW, I think about these kinds of things as *protocol* analysis. Yeah, there’s packets here, but it’s really the protocols that matter, not so much the individual packets. If you come to recognize the bunch of individual packet events involved in TCP flows or PIM-sparse setups as the beautiful ballet that they are, then you know that you’re on the right track. And then *also* watch that girl on the obstacle course. Amazing.

Reply
Kary July 18, 2014

Wow, great story, Chris!

> Wrestling the fix from the OS vendor by myself made this victory particularly sweet, earned me a ton of capital to do my own thing, and led to the most interesting problems showing up on my desk.

THIS IS WHAT I’M TALKING ABOUT, PEOPLE. You can’t get this kind of win without packet level analysis. And more capital for yourself makes you more valuable. Companies will usually pay to keep their valuable employees and if they won’t, someone else will.

You’re right; what I’m talking about is packet level analysis of protocols at all layers of the stack. I think people understand what I’m getting at but maybe I should rethink my terminology.

Reply
Peter Higgins July 19, 2017

Great posts here Kary.

Where are you located?
Are you a Packet Analyst with an Organization or are you an Indy?

Peter

Reply
    Kary July 19, 2017

    Hi Peter, I’m in the Bay Area in California. I guess you could say I’m an indy but I don’t offer a paid service at the moment

    Reply
Add Your Reply

Leave a Comment: