Network "Ghostbusting" with isaac

One of the worst network issues to troubleshoot is the "ghosts in the network". This is the kind that are random, sporadic and as soon as you look one place, they seem to move. You know what I mean right?

It starts with a call or email "Hey the network seems slow" and as you start looking at the network closet the user is in, you hear another "Is the Internet down?" from the other side of the building. Next more calls come in but everything you monitor shows up and working fine. Usually these come in at lunch, or even worse just as you sit down to a nice steak dinner at a fancy restaurant...

You can spend a lot of time tracking down these issues, at least I know we do, well used to. I'm pretty lucky because when we are troubleshooting an issue, we usually will grab a few of our network engineers to help us. Not because we can't figure it out, but it lets them see what we do to troubleshoot and often times leads to a new idea for a product. Flow setup throttling came up this way a few years back and some cool isaac commands started this way too.

So imagine you start hearing about ghosts. What can you do. The first thing is to find the user. There's not much point troubleshooting the Andover LAN, when the user is travelling and actually in Germany. (Trust me, been there before). This will get you all the technical details you need, like which port or access point they are connected to, what the IP is, what role they are in etc. It also gives you a regional and local map in case you need to dispatch someone out there. The command is called "find ", pretty obvious right? If not you can actually create a new name for the command like locate, or "vinden" if you want the command in Dutch.

The next thing is "topintferrs" this will show you the ports in your network that are generating the most errors. This could point you to a bad GBIC or fiber connection or a bad copper cable. I also usually run "topusers" which will show the ports with the most traffic on them. If a link is running close to 100%, start looking there.

If you find a bad cable, you have the information to send the closest technician out to fix it. If you find it's a single users saturating a link, you can issue "ratelimit" to slow them down, or "blacklist" to kick them off of the network entirely.

The best thing is using isaac, you can do this from your iphone, while at dinner without leaving the table and letting your food get cold....