How to Create a Server Failover Solution

How to create a server failover solution

Posted on May 16, 2013 by Shane Helpton Posted in Web Servers

An automatic server failover solution can prevent your website from going down in the event of a server failure. Through automatic detection, an error on your primary server can be detected and traffic will automatically be sent to a backup server.

Overview

An automatic server failover solution is surprisingly easy to setup. It works by having two servers with identical content on them – a primary server and a secondary server. A third server monitors the primary server and detects if there is a problem. If there is, it will automatically update the DNS records for your website so that traffic will be diverted to your secondary server. Once your primary server is functioning again, traffic will be routed back to your primary server. Most of the time your users won’t even notice a thing!

Setting up an automatic server failover solution just consists of 5 steps:

1. Get a secondary server
2. Synchronize primary and secondary servers
3. Reveal server status
4. Set up DNS Failover
5. Test it!

Step 1: Get a secondary server

The first thing you need is another server to be your secondary/failover server. This doesn’t need to be a dedicated server – if you’re using shared hosting then another shared hosting account will do just fine. However it is important that you choose a server that is physically different from your primary server. I recommend not only using a different hosting company, but also making sure that the servers are in different locations.

What good does a secondary server do if its using the same network as your primary server when it goes down? Or if a server admin makes a goof that’s propagated to all servers they manage? Before I sign up with any new host I’ll research where they’re located to make sure it’s not in the same state as my primary server. You can use the IP Information tool here to check the location of any ip address

If you don’t know where to find a good secondary server host, let me suggest StableHost. I recently moved over to them after too many problems with HostGator, and I’ve been really impressed. They are the best bang for your buck – great plans with realistic prices (as of now, shared starting at $5/month). Plus they just migrated all their servers to a Provo, Utah location, which I doubt is the location of your primary server. Oh, and here’s a StableHost coupon code for 40% off: expert40

Step 2: Synchronize primary and secondary servers

Right now you have a blank backup server, but sending people there in the event of an emergency will do no good unless it mirrors your primary site. You could copy everything over right now, but then you’d have to manually update the secondary server every time you change the primary server, which is a hassle. What we need next is a way to automatically keep your primary and secondary servers in sync with each other.

Syncing website files

There are two common approaches I typically use to keep my website files in sync: Rsync and Source Code Repositories.

Rsync is a unix utility that synchronizes files and directories from one location to another. You run a simple command similar to “rsync server1:/myfiles server2:mybackupfiles” and it will copy everything inside ‘myfiles’ from server1 into the ‘mybackupfiles’ folder on server2. It also saves time and bandwidth by only copying over what is different, pretty cool!

You’ll run a command similar to this on your secondary server. Be sure to use the IP address of your primary server instead of the host name since the host name could refer to the secondary server when failover has kicked in (that’ll make sense later). I’d suggest mirroring everything in the public html directory.

rsync -avz email:/home/mywebsite/public_html /home/secondarywebsite/public_html

In order for this to work, you need to set up ssh keys so that the secondary server is allowed to access the primary server (security!). This tutorial can help you set that up: Setting up ssh keys for rsync. It looks daunting, but really isn’t that bad. In that article, ‘thishost’ is the secondary server, and ‘remotehost’ is the primary. You can skip the ‘validate-rsync’ part if you wish.

If you can run that rsync command successfully via command line, it’s time to automate it so that it runs without you having to run it. I configured mine to run every night at midnight, but you can choose to make yours run more or less often, depending on how often your server files change. We’ll be automating this with a cronjob. Type ‘crontab -e’, and then add this line, where ‘rsync…’ is your rsync line:

0 0 * * * rsyc...

An alternative method is to use a source code management system and pull from it regularly. For example, if you have a git repository set up that contains all of your website files, simply running a ‘git pull’ instead of that rsync command will accomplish the same by pulling the latest files from the repo. Here’s a tutorial for Setting up git on a server.

Syncing databases

If your website replies upon a database, such as wordpress does, then you’ll also need to copy over the database, which is separate from the file copy you’ve just set up. Technically you could try to just copy over the database files, but I wouldn’t recommend that – it could create inconsistent data in the event of an error. I’ll show you how to copy over the contents directly from within your database program.

We’ll use MySQL for this example, but all database systems should have similar commands. You’ll first need to have a blank database set up on the secondary server with the same name, user, and password as the primary server’s database. Then we’ll utilize the mysqldump utility to spit out the contents of the primary server’s database and feed it directly to the copy. This command should be run from the secondary server:

mysqldump --host=1.2.3.4 --user=MYDBUSER -pMYDBPASSWORD --add-drop-table --no-create-db --skip-lock-tables MYDBNAME | mysql --user=MYDBUSER -pMYDBPASSWORD MYDBNAME

Be sure to replace the values above with your actual primary server ip address and user/passwords. Once you’ve verified that this command works, simply run it as a cronjob as we have done with rsync or git. Run ‘crontab -e’, then add your mysqldump command:

0 0 * * * mysqldump...

Now your database will be mirrored onto your secondary server every night! This solution works well for websites that don’t absolutely need their secondary server to have an exact up-to-date copy of their primary data. Understand that if you add a new wordpress article during the day and then your server fails over to the secondary, the secondary won’t have that new article in its database until the database sync is run again at night. If you need a real-time copy of your data, you can look into setting up a master-master replication solution, or come up with a manual method of copying over a subset of the data as it is changed given you know how your website works.

Most web applications depend on your website files and database to run smoothly, which we have taken care of. If your server depends on other functions, such as mail, you’ll have to come up with a plan to keep those systems in sync as well. Unfortunately that is beyond the scope of this tutorial.

Step 3: Reveal server status

The tool we’re going to use to automatically switch your servers upon failure needs to know when your server is failing. It does this by checking your server every few minutes for a specific response. You can have it check just your homepage to see if its serving content, but I like to get a little more detailed. For example, your website may still return a page if your database is down, which would be a false indicator that everything’s okay when it’s not.

I recommend creating a server status page that reveals if ALL services are functioning properly. Make a simple script that connects to the database and returns ‘SUCCESS’ only if the connection was successful. This ensures that HTTP and database services must be functioning properly to report a successful server status. If there are other services your site depends on, include a check for those too. Here’s a sample PHP script I use to report that my server is healthy:

//connect

$link = mysqli_connect("localhost", "my_user", "my_password", "my_db");

// check connection

if (mysqli_connect_errno()) {

printf("Connect failed: %s\n", mysqli_connect_error());

exit();

}

//perform simple query

if ($result = mysqli_query($link, "SELECT 1")) {

if(mysqli_num_rows($result)) echo "SUCCESS";

mysqli_free_result($result);

}

//close

mysqli_close($link);

After that file is set up, you should be able to go to that page in your browser and see SUCCESS!

Step 4: Set up DNS Failover

We’ll be using a service called DNS Made Easy to provide DNS Failover. This automatically checks your server status script every few minutes for ‘SUCCESS’. In the event your server is unreachable, or there’s some other error and it doesn’t see ‘SUCCESS’, DNS Made Easy will change your DNS entries to point to your backup server. As soon as your primary server is reachable again DNS Made Easy will revert the changes back to your primary server . Since your visitors are accessing your site via your domain name, no one will ever know what just happened as they’ll have been pointed to a functioning server the whole time!

You’ll need to purchase a DNS Made Easy account and then point your DNS their name servers so they are handling your domain name. Their Business level account ($60/year) includes up to 3 hostnames to monitor (you’ll need 2: one for and one for Note that this price is per year, not per month – pretty cheap considering the peace of mind it provides!

After you create your account and set up your basic name server settings, you’ll need to configure the automatic failover. In the A-records table of hosts, click the Off in the SM/FO column and input your values:

Use these settings:

• Sensitivity: High so that your server will be checked more often and routed quicker in the event of an outage.
• File To Query: this is the path to your server status script that should return ‘SUCCESS’. Use a relative path (‘/check/status.php’) that doesn’t include your domain or IP address.
• String To Query For: SUCCESS. Or if you used a different message in your server status script.
• IP Address 1: This is the IP of your primary server
• IP Address 2: This is the IP of your secondary server
• Monitoring Notifications and DNS Failover: make sure these are checked

Click OK to save – it should now read ‘On’ in the SM/FO column next to that host. Then do the same for the hostname with no name next to it (the blank one is for people that access your site without the www in front).

One last thing you’ll need to do is set the TTL to 300 seconds. TTL is the Time To Live, and it means how long DNS entries should be cached for. The lower the value the quicker DNS will be updated for those that have already been to your site. It can be set as low as 60 seconds, but 300 is more likely to be honored by big ISPs who would otherwise use a much larger default value – plus it lowers our queries per month. Set this with the Edit icon while your A-record is checked in the table.

Step 5: Test it!

Make sure you wait at least 2 days before testing this – it will take 24-28 hours for your initial DNS transfer to DNS Made Easy to propagate through the internet. To test, simply rename the server status script so that the monitoring cannot find it. Within 2-4 minutes, you’ll receive an email alerting you that your site is down (you’ll actually receive 2, one for each host). Once this happens, ping your website – it’ll show you the IP address mapped to your domain name.

You should see your secondary server’s IP address. If you don’t, it’s because you recently visited your site and the entry was cached. You’ll just have to wait a maximum of 300 seconds (that TTL setting) for it to expire and your computer to make a new lookup. If your ping result doesn’t show a different IP address within 5 minutes, check over all the steps to make sure it’s been set up properly

Tips

• In DNS Made Easy‘s settings, you’ll notice that there are 5 IP addresses listed. This allows you to string up to 4 backup servers! It will keep checking down the chain of servers until it finds one that works, just in case your first secondary server isn’t working either.
• If your primary server is down for a long time, you might want to disable your automatic server syncing so that your active secondary server doesn’t get interrupted while handling live traffic
• You can utilize this approach to help with major server maintenance. Simply fake an outage by renaming the server status script. Traffic will be diverted to your secondary server and you can perform major updates on your primary server. When you’re done, revert the server status!

Summary

You’ve successfully created an alternative server, synchronized it to be an exact copy of your primary, and set up failover so in the event of a disaster it will automatically kick in to serve your traffic. Next time your primary server fails you won’t have to do a thing – and you didn’t have to install any failover software! You can rest assured knowing your visitors are able to access your website at all times. Congratulations!