Switch Troubleshooting for Fun and Profit

I spent an entire day troubleshooting odd network problems at a local client last Friday. The symptoms were:

  • Users could not
    • get their email
    • access files on the server
    • log on successfully
  • Internet access was sluggish, for some users, but fine for others.
  • Software deployments from across the networks (content filtering software, Ghost images, GPO-deployed packages) would not deploy.
  • Ping times from a workstation to a server (across 5 switches) ranged anywhere from <1ms to several thousand ms, or just plain timeouts.
  • Ping times from a workstation to another workstation, connected on the same switch, were only marginally better, resulting in anything from 1ms to frequent timeouts and dropped packets.
  • Ping times to the local switch, to which the servers were plugged in, ranged from 1ms to several hundred ms.

Several hours of troubleshooting resulted in the following highlights:

  • After two hours of capturing packets, 30% of all network traffic was ARP. This is with only a handful of desktop machines powered on. Not good.
  • Rebooting one switch made a huge difference, and all of a sudden network traffic was working again. However, things were still slow on occasion.

So tomorrow I will be making some VLANs to cut down on the broadcast traffic across these eight or so switches. Fun!

Leave a Reply