This is an article that helps you to perform a server migration with zero downtime for the sites. This is applicable for cPanel linux servers. Important things that should be checked before doing a server migration are:
Pre Migration checklist
#Connectivity issue of both servers
This includes the SSH port, wheel user configuration, authentication type, public keys, allow and deny lists, allow users etc.
# Compatibility of application versions and enabled modules
The compatibility of PHP/MySQL versions, PHP modules and type, MySQL conf, Apache version, etc. should be checked
# Disks and the partitions in the new server
Number of partitions and disk availability comparison, RAID configurations, secondary disks, mount options are checked in this step.
# IP address availability in new server
Number of used IP addresses ( including the nameservers, gateway, broadcast, host, dedicated IP address ) in the new server are checked.
Once you have made sure about the above configurations, you can start with the steps to perform the ‘zero downtime migration’. Below are the steps that you should follow.
(** Note: If you are doing the syncing process using a script, then copying the public_key to the server will help you in saving time from entering the user password each time)
Migration checklist
1. Reducing the TTL value for the domains and the nameservers
Reduce the TTL value for the zone files to a value 300 or 150 using a script and then reload rndc to update the changes. This is done to switch back the accounts to the old server if you find any issues in the new server. If the TTL value is not reduced, then it can cause a downtime which will be equal to the TTL value in seconds while switching back the accounts. Below is a sample script to change the TTL for the zone files.
| for i in $(find /var/named/ -maxdepth 1 -type f -iname “*.db”); do sed -i -r ‘s/$TTL [0-9]{3,5}/$TTL 300/g’ /var/named/$i ; done |
2. Checking and making the list of domains that are resolving to any external servers
This is to check and confirm the domain status in the new server.
3. Doing the migration of the accounts to the new server
Using the option “Copy multiple accounts/packages from another server” in WHM, migrate accounts from the old server to new server. Migrating all the accounts at a time is not recommended. Connection loss issues can cause skiping of accounts. Migrating 10 to 20 accounts at a time is recommended. Collect the account usernames that are migrated successfully to the new server. This can be used for re-syncing data after A record change.
4. Change the A record for the domains in the old server to that of the new server
Change the A record for the migrated sites including the add-on domains/parked domains to that of the new server. You can use script to do this. A sample script that will help in changing the IP is as below.
A script to get all the domains including the main domains, add-on domains and parked domains.
| while read i; do dom=$(grep $i /etc/trueuserdomains | cut -f1 -d:); grep $i /etc/userdomains | awk -F: ‘{print $1}’ | egrep -v “^.+.$dom”; done </tmp/migrateuser >/tmp/migratettl |
A script to change the IP from 0.0.0.0 to 2.2.2.2 for all the domains.
| while read i; do sed -i -r ‘s/0.0.0.0/2.2.2.2/g’ /var/named/$i.db; done</tmp/migratettl && rndc reload |
5. After the propagation of the TTL, re-sync the databases, mails and site contents to the new server
Do the re-syncing of mails, site contents and databases to the new server. This is done to ensure 100% of data is transferred to the new server. All the updates/changes made after the migration process till the A record change will be synced to the new server. Before database is synced, make sure that you have suspended the account or stopped the mysql service in the old server. This is because we are re-syncing the “/var/lib/mysql/$username_db/” folder. If the mysql access is not ceased, then it can create incomplete tables in the new server. Sample scripts to do this is given below.
A script to re-sync all the updates including the mails and the files. “/tmp/migrate” file should contain the migrated usernames.
| while read x; do rsync -rave”ssh -l root -p $PORT” /home/$x/ $REMOTE_SERVER:/home/$x/ ;done </tmp/migrate |
A script to collect the databases and then sync the contents.
| for i in $(cat /tmp/migrateuser); do ls /var/lib/mysql/ | grep $i | cut -f1 -d/ ;done >/tmp/migratedatabase while read d; do rsync -rave”ssh -l root -p $PORT” /var/lib/mysql/$d/ $REMOTE_SERVER:/var/lib/mysql/$d/ ; done</tmp/migratedatabase |
6. Make sure the sites and mails are working fine
Make sure that the re-synced sites and mails are working fine. You can check the apache access logs, mail logs, apache error logs, etc.. to make sure this. Also, dig the domain to see if the domain is loading from the new server or not.
7. Update the license for software like WHMCS, kayako etc. with the new IP address
Software like WHMCS, kayako etc.. have their license set with the IP address of the server where the domain is hosted. If any such software is present in the server, make sure you have updated the software license in the new server with the new IP address. Most of such software has its portal to do this.
8. Create the zone files for the same nameservers with the new IP in new server with reduced TTL value
Create the zone files for the nameservers in the new server with the new set of IP’s. Once it is created, reduce the TTL to 300 or below and reload the rndc. For cPanel, use the below script to add the dns for the nameservers. Instead of $DOMAIN, use the nameserver name.
| /scripts/adddns –domain $DOMAIN –ip $IP |
9. Change the IP address in the old server for the nameservers to that of the new server
When all the accounts are migrated and re-synced. Change the A record for the nameservers to that of the new set and check if the nameservers are resolving to the new IP. Also, confirm that the new server’s named is receiving and acknowledging the requests.
10. After resolving, change the IP address for the nameservers from domain management
When you are sure that the nameservers are working fine and the new server performances are good enough, do the IP address change for the nameservers from the domain management (registrar level). Note that this will take 24 hours to get propagated. So, do not make this change if you have any confusion with the status of the new server.
11. Switch-off the services in the old server after making sure the log update
Once all the migration is done and made sure that the sites are fine. Switch-off the services except the named in the old server. Named should not be stopped because there are possibilities for handling the requests by the old server.
12. Double check the account list in both and make sure the migration is done for all the accounts
Once this is done, double check the number of accounts in both the servers. This is to find if there are any accounts created in the old server after the migration. If there are accounts like that, migrate those to the new server and re-sync the data.
The above is a very rough outline of the Data Migration service we provide, and if you have any questions, we would be happy to talk to you! 🙂
About the Author :
Sambhu PS works as a Software Engineer in Bobcares. He joined Bobcares back in February 2011. He loves reading technical blogs, plays violin plays, table tennis and listens to music in his free time.
Blog edited by :
Kumar K works as a Junior Software Engineer in Bobcares. He joined Bobcares back in February 2012. He loves reading books, watching movies and listening music in his free time.
database sync means both mysql servers are down! where is zero downtime?