Yesterday we had an issue with one of our 100 megabit connections to the outside world, which runs most of our WAN traffic. I’m posting this to try and convince ya’ll on the interwebs to double-check my thinking.
So, here’s the timeline:
- Approximately 1305 outbound usable bandwidth from our active GWIP link dropped to an average of ~20Mbit/second. The maximum outbound packet rate sat on approximately 5000 packets/second. Inbound data was not affected.
- Investigations started at approximately 1330, it was originally believed that power brownouts which affected many sites in the SEQ region at approximately 1215-1220 could have been affecting services.
- At approximately 1415pm a job was logged with The Vendor.
- The job was escalated within The Vendor to Tier2 at approximately 1500, and I spoke with some Vendor reps about the escalation approximately ten minutes later.
- As of approximately 1515, packet rates started to rise and outbound bandwidth started to increase.
- I spoke with their tech at approximately 16:15; he advised that during the impact to services we were over-utilizing the link.
- The Vendor claims that due to the fact that their systems show 5 minute averages and ours 1 minute, this explains the 80Mbit/second disparity in traffic displayed on our respective systems.
Other facts:
[Read More]

