Getting HEAD

Crass, I know, but it’s part of my charm. We run load balancers for our proxies and we need to monitor they’re working, so we make requests through them out to the internet. I was doing some checking one day on “top utilization by user” and found that the monitoring account had been… rather hungry.

Turns out, we were doing a full GET request to two news sites, for every time a proxy appeared in a load balancing pool, from each of the load balancers.

  • Two Load Balancers
  • Two Sites
  • Eight Proxies
  • Sixteen pools
  • Every five seconds

2 x 2 x 8 x 16 = 512 requests every five seconds, or 6144 a minute!

Given every page load was anywhere up to half a megabyte depending on the code they decided to inline, this was just excessive (and rude, to the sites we were hitting).

So I changed it to a HEAD request.


We still needed the calls to make sure the end-to-end connectivity worked, but we dropped from 55 MB/minute to 2 MB/minute, with no loss in functionality. Amazing, hey?

#proxy #graphs