A structured troubleshooting method

Why use a method?

When users shout that "the network is down", it is tempting to start changing things: reboot the switch, change a cable, edit an ACL. Random changes waste time, can create new faults, and even when the problem goes away you do not know which change fixed it. A method keeps you calm and gives your manager clear updates.

The steps

1. Define the problem: who is affected, what exactly fails, since when, and what changed. "Billing PCs on floor 2 cannot open the billing server since 10:15; other floors are fine" is a good definition; "network slow" is not.
2. Gather facts: ask users, look at the network diagram, run ping and show commands, read the logs.
3. Analyse: compare the facts with how the network should work (the baseline and the documentation).
4. Eliminate: rule out what is clearly working. If the PC can ping its gateway, the cable, the access port VLAN and the PC's IP settings are probably fine.
5. Propose a hypothesis: the most likely cause, for example "the trunk to the floor 2 switch does not carry VLAN 30".
6. Test the hypothesis: check it with a command, then make ONE change. If it does not help, undo it and test the next hypothesis.
7. Solve: apply the fix, confirm with the users that everything works, and save the configuration.
8. Document: write down the symptom, the cause, the fix and the time, and update the diagram if it changed.

Where to start: the approaches

Divide and conquer is popular because one ping answers a lot. If a PC can ping a remote server by IP, layers 1 to 3 work end to end, so the problem is higher up, perhaps DNS or the application. If the ping fails, the problem is at layer 3 or below, and you move down.

Worked example: the hospital billing PCs

At a hospital in Lucknow, the six PCs at the billing counter cannot reach the billing server at 10.10.50.20. The pharmacy and the wards can.

• Define: only billing PCs (VLAN 30, 10.10.30.0/24) are affected, since this morning. A change ticket shows the floor switch was replaced last night.
• Gather facts: the PCs have correct DHCP addresses and gateway 10.10.30.1. ping 10.10.30.1 fails from every billing PC. A PC in the pharmacy VLAN on the same switch works.
• Eliminate: one PC could have a bad cable, but six PCs failing together points to something they share: the VLAN path.
• Hypothesis: the new switch's uplink trunk does not allow VLAN 30.
• Test: show interfaces trunk on the new switch lists VLANs allowed: 1,10,20,40. VLAN 30 is missing.
• Solve: switchport trunk allowed vlan add 30 on the uplink. The PCs can now ping the gateway and open the billing software. Save with copy running-config startup-config.
• Document: cause, fix and a note to add the VLAN check to the switch replacement checklist.

Baselines and documentation

A baseline is a record of what normal looks like: normal CPU load, link usage, number of OSPF neighbours, response times. Without it you cannot tell whether 70 percent CPU is a problem or just Monday morning. Documentation includes a network diagram (physical and logical), an IP addressing plan, VLAN lists, and saved copies of the configurations. When the diagram is out of date, every fault takes longer to find.

Change management

Many outages start with a change. Change management is the process for making changes safely: write down what will change and why, get it approved, do it in an agreed window (often at night), have a rollback plan, and test afterwards. When something breaks, the first useful question is always "what changed?".