May First/People Link has several members that are targets of politically motivated denial of service attacks (mostly groups that support reproductive justice for women and palestinian rights). To fight off the attacks, we work closely with Deflect - a non-governmental organization based in Canada that fights against this kind of censorship.
When a site is down, it's not always easy to understand why. Deflect runs as many as 5 edge servers, any of them could be down. And, of course, the origin server could also be down.
I tried using a commericial/free as in beer service for monitoring up time, but when it reported the site being down, I had no idea which part was down.
Here are few of my tips.
First, you usually want to lookup the IP addresses reported by DNS to you can get a list of your edge servers:
dig +short mayfirst.org
Then, you can use httping to test each one, e.g.:
httping -l --divert-connect 220.127.116.11 https://mayfirst.org/
The -l says to use https (seems redundant if you specify https as the target, but if you don't put -l, httping will complain).
The --divert-connect option is particularly useful with your origin IP - because you can compare response times and ensure that your origin is up.
This is all fine and good for generally testing your edges, but sometimes it's a particular page that is causing the problem. We've found that edges sometimes cache error pages, even after the error has gone away. But... how can you prove it?
curl --resolve mayfirst.org:443:18.104.22.168 --head https://mayfirst.org/en/contact/
The --resolve argument is the key here - it allows you to specify how a domain should be resolved (with this option, curl does not perform a DNS lookup). Replace the IP address with the origin IP address and you have a nice comparison between the edge and the origin. The caching server should respond with a Via header to confirms it is caching. And, of course, the first line shows you the http response which should be the same on the origin and caching server.
I've written the script below to help debug sites that may not be working properly. It combines both the httping and curl options.
a-reverse-proxy-test domain.org [origin-ip]
Here's the code:
#!/bin/bash # Test all given edges using httping and curl. Pass the full protocol, domain # and page you want to test as the first argument. Optionally pass the origin # IP address as the second argument (so you can tell if your origin is down). # For example: # # a-reverse-proxy-test https:/domain.org/contact 22.214.171.124 # url="$1" origin="$2" if [ -z "$url" ]; then printf "Please pass the url as first argument.\n" exit 1 fi proto=$(echo "$url" | sed -E "s#(https?).*#\1#") host=$(echo "$url" | sed -E "s#https?://([^/]+)/?.*#\1#") if [ -z "$proto" ]; then printf "Please include http or https as part of the url.\n" exit 1 fi if [ -z "$host" ]; then printf "Failed to extract the host. If passing just host, ensure it ends in a /, e.g. https://mayfirst.org/.\n" exit 1 fi if ! ping -c 1 126.96.36.199 >/dev/null; then # printf "We are offline. Not running.\n" exit 1 fi ips=$(dig +short "$host") if [ "$?" -ne "0" ]; then # printf "DNS lookup failure. Not running.\n" exit 1 fi if [ -n "$origin" ]; then ips="$ips $origin" fi l= port=80 if [ "$proto" = "https" ]; then l=-l port=443 fi for ip in $ips; do printf "Testing IP: %s\n" "$ip" printf "httping response: %s\n" "$out" httping $l -m -t 5 -c 1 --divert-connect "$ip" "$url" printf "curl HTTP and Via headers:\n" curl --resolve "$host:$port:$ip" --head "$url" 2>&1 | egrep "^(HTTP|Via)" printf "\n" done
To help with regular monitoring (using
httping) I wrote a script that runs via a cron job and outputs the results to a log file.
#!/bin/bash # Test all given edges domain="$1" origin="$2" proto=http if [ -n "$3" ]; then proto="$3" fi if [ -z "$domain" ]; then printf "Please pass the domain as first argument.\n" exit 1 fi if ! ping -c 1 188.8.131.52 >/dev/null; then # printf "We are offline. Not running.\n" exit 1 fi ips=$(dig +short "$domain") if [ "$?" -ne "0" ]; then # printf "DNS lookup failure. Not running.\n" exit 1 fi if [ -n "$origin" ]; then ips="$ips $origin" fi l= if [ "$proto" = "https" ]; then l=-l fi for ip in $ips; do date=$(date +%Y.%m.%d-%H:%M) for i in 1 2 3; do out=$(httping $l -m -t 5 -c 1 --divert-connect "$ip" "$proto://$domain") [ -z "$out" ] && out=1 printf "%s %s %s\n" "$date" "$ip" "$out" done done