Crass, I know, but it’s part of my charm. We run load balancers for our proxies and we need to monitor they’re working, so we make requests through them out to the internet. I was doing some checking one day on “top utilization by user” and found that the monitoring account had been… rather hungry.
Turns out, we were doing a full GET request to two news sites, for every time a proxy appeared in a load balancing pool, from each of the load balancers.
- Two Load Balancers
- Two Sites
- Eight Proxies
- Sixteen pools
- Every five seconds
2 x 2 x 8 x 16 = 512 requests every five seconds, or 6144 a minute!
Given every page load was anywhere up to half a megabyte depending on the code they decided to inline, this was just excessive (and rude, to the sites we were hitting).
So I changed it to a HEAD request.
We still needed the calls to make sure the end-to-end connectivity worked, but we dropped from 55 MB/minute to 2 MB/minute, with no loss in functionality. Amazing, hey?