Getting Closer
My master nameserver is on vps3. My outgoing SMTP server is now on vps3. My inbound smtp server is now split between vps2 and vps3. Varnish cache is now installed on vps3 and vps2.
Installing and configuring nginx is going to be almost as complex as installing and configuring the vps itself because of the number of things that need to go right for everything to work.
For example, if I make a typo with an IP address then a Web site might not work and finding that typo might take a lot of time.
This is, perhaps, where my ULA network can come to the rescue. Moving from vps2 to vps3 should be as simple as installing nginx; copying all the relevant configuration files, TLS keys, and certificates across; adding the public IPs to the interfaces; adding the public IPs and ports to the firewall; modifying the public (HTTP/HTTPS termination) IP settings for nginx, so they are vps3 IPs; modifying the private (ULA network) IP settings in nginx, so they point to the vps3 varnish instance; restarting nginx; modifying the DNS A and AAAA records in NSD and Cloudflare.
Not that it will be that simple, however.
Some of the Web sites I host use MySQL, so I will need to export the databases from vps2 to vps3, ensuring I make the databases on vps2 read only until the move is complete so there are no new records created that get lost after switchover.
Future Changes After Transition
DNS
At the moment DNS requests (excluding domains using Cloudflare) are made to 3 destinations: vps2 (2 DNS servers), vps3 (1 DNS server), and Hurricane Electric (4 DNS servers).
Something I am interested in doing is moving from Hurricane Electric to Esgob Limited for secondary DNS.
While Hurricane Electric helped me (and many others) understand IPv6, their free DNS service does have limitations. Some of those limitations are imposed (e.g. number of zones, number of records per zone) while others are due to lack of support (e.g. still no DNSSEC support).
If I make such a transition for secondary DNS provider, I will be moving from 7 unicast DNS servers to 1 unicast and 1 anycast.
Installation
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 00A6F0A3C300EE8C
sudo nano /etc/apt/sources.list# nginx
deb http://ppa.launchpad.net/nginx/development/ubuntu trusty main
deb-src http://ppa.launchpad.net/nginx/development/ubuntu trust mainsudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install nginx
sudo apt-get autoremove
sudo apt-get autocleanConfiguration
As with Varnish, configuring nginx should be as simple as copying across the nginx configuration files, the SSL certificates and keys, and changing the IP addresses.
sudo su
rsync -avrlogpP --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' thejc@vps2.thejc.me.uk:/etc/nginx/ /etc/nginx/
rsync -avrlogpP --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' thejc@vps2.thejc.me.uk:/etc/lighttpd/ /etc/lighttpd/
rsync -avrlogpP --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' thejc@vps2.thejc.me.uk:/etc/ssl/ /etc/ssl/
exit
scp -P 8043 thejc@vps2.thejc.me.uk:~/Scripts/ocsp-stapling.sh ~/Scripts/
scp -P 8043 thejc@vps2.thejc.me.uk:~/Scripts/ocsp-stapling-cache-now.sh ~/Scripts/
sudo crontab -e30 2 * * Mon,Wed,Fri,Sat /home/thejc/Scripts/ocsp-stapling-cache-now.shsudo ~/Scripts/ocsp-stapling-cache-now.sh
sudo mkdir -p /var/cache/nginx
sudo chown www-data:www-data /var/cache/nginx
sudo service nginx restartIf everything is working as it should at this point (i.e. the OCSP stapling script can verify all the current certificates as good) then restarting nginx should fail.
The only errors timestamped at the most recent attempt at restarting nginx should be about problems binding to a requested IP address. It is now time to modify the IP addresses in the nginx configuration files on vps3.
This is going to be rather simple. Search for the string 2001:470:1f09:38d::80: and replace it with 2a03:ca80:8001:769d::80:.
cd /etc/nginx/
sudo su
find . -type f -print0 | xargs -0 sed -i 's/2001:470:1f09:38d::80:/2a03:ca80:8001:769d::80:/g'A recursive search of strings 2001 and 2a03, however, show that some issues remain. On my previous VPS I had two IPv6 IP addresses, so I need to do a recursive delete of lines containing 2a03:ca80:8000:.
find . -type f -print0 | xargs -0 sed -i '/2a03:ca80:8000:/d'Likewise, I need to make a similar change for IPv4 IP addresses. On my previous VPS I had 3 different IP(v4) addresses and one my new one I only have one (technically two, but it should be one at the start of next month as I've dropped the additional IP address).
Thus I need to search for 149.255.99.49, 149.255.99.50, and 149.255.97.82 and replace them all with 149.255.108.141.
find . -type f -print0 | xargs -0 sed -i 's/149.255.99.49/149.255.108.141/g'
find . -type f -print0 | xargs -0 sed -i 's/149.255.99.50/149.255.108.141/g'
find . -type f -print0 | xargs -0 sed -i 's/149.255.97.82/149.255.108.141/g'
exit
sudo service nginx restartA final configuration issue: I haven't searched and replaced ULA IP addresses.
find . -type f -print0 | xargs -0 sed -i 's/fdd7:5938:e2e6:9660::80/fdd7:5938:e2e6:6c8d::80/g'The final thing to do is disable mail3.thejc.me.uk:443 and [::]:80 on vps3, and to add the IP addresses for the Web sites that are currently enabled.
cd /etc/nginx/sites-enabled/
sudo rm default
sudo rm mail3.thejc.me.uk
grep -R "\[\(.*\)\]" ../watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [fdd7:5938:e2e6:6c8d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./watfordjc.co.uk: resolver [fcf4:90db:f24c:72ca:df4d:b9ee:be0b:c37d] [2001:470:1f09:38d::fcf4:53] [2001:470:20::2] 74.82.42.42 8.8.8.8; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [fdd7:5938:e2e6:6c8d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./watfordjc.co.uk: resolver [fcf4:90db:f24c:72ca:df4d:b9ee:be0b:c37d] [2001:470:1f09:38d::fcf4:53] [2001:470:20::2] 74.82.42.42 8.8.8.8; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:443 ssl spdy; ./watfordjc.co.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./watfordjc.co.uk: resolver [fcf4:90db:f24c:72ca:df4d:b9ee:be0b:c37d] [2001:470:1f09:38d::fcf4:53] [2001:470:20::2] 74.82.42.42 8.8.8.8; ./health.thejc.me.uk: listen [2a03:ca80:8001:769d::80:4]:80; ./gpg.thejc.me.uk: listen [2a03:ca80:8001:769d::80:a]:80; ./david-j-lane.co.uk: listen [2a03:ca80:8001:769d::80:3]:80; ./burnthefat2014.thejc.me.uk: listen [2a03:ca80:8001:769d::80:1]:80; ./blogs.thejc.me.uk: listen [2a03:ca80:8001:769d::80:4]:80;
sudo nano /etc/init/ipv6-eth0.conf…
pre-start script
…
/sbin/ip -6 addr add 2a03:ca80:8001:769d::80:1 dev eth0
/sbin/ip -6 addr add 2a03:ca80:8001:769d::80:3 dev eth0
/sbin/ip -6 addr add 2a03:ca80:8001:769d::80:4 dev eth0
/sbin/ip -6 addr add 2a03:ca80:8001:769d::80:a dev eth0
/sbin/ip -6 addr add 2a03:ca80:8001:769d::80:c dev eth0
initctl emit ipv6-eth0-ips
end script
pre-stop script
…
/sbin/ip -6 addr del 2a03:ca80:8001:769d::80:1 dev eth0
/sbin/ip -6 addr del 2a03:ca80:8001:769d::80:3 dev eth0
/sbin/ip -6 addr del 2a03:ca80:8001:769d::80:4 dev eth0
/sbin/ip -6 addr del 2a03:ca80:8001:769d::80:a dev eth0
/sbin/ip -6 addr del 2a03:ca80:8001:769d::80:c dev eth0
end scriptsudo nano /etc/init/ipv6-lo.conf…
pre-start script
…
/sbin/ip -6 addr add fdd7:5938:e2e6:6c8d::80:1 dev lo
/sbin/ip -6 addr add fdd7:5938:e2e6:6c8d::80:c dev lo
initctl emit ipv6-lo-ips
end script
pre-stop script
…
/sbin/ip -6 addr del fdd7:5938:e2e6:6c8d::80:1 dev lo
/sbin/ip -6 addr del fdd7:5938:e2e6:6c8d::80:c dev lo
end scriptsudo rebootAfter a reboot, everything should be functioning. Two of the sites that are functioning on vps2 use MySQL, so I still need to do some work on getting those working, and two of the resolvers listed are located on vps2 so I still need to add a recursive DNS server on vps3.
Another thing I have yet to do is to allow traffic to nginx through the firewall. That is a simple case of adding the IPv4 and IPv6 IP addresses to my ip6tables.save and iptables.save files and accepting incoming connections to those IPs on ports 80 and 443.
Testing
Back on vps2, I can now test my active sites on vps3 are working properly. First, I need to install gnutls:
sudo apt-get install gnutls-binWith gnutls installed, I can now test that watfordjc.co.uk and johncook.co.uk are working:
gnutls-cli -p 443 2a03:ca80:8001:769d::80:1 --insecure
GET /news.php HTTP/1.1
Host: web.watfordjc.co.uk
HTTP/1.1 502 Bad Gateway …
gnutls-cli -p 443 2a03:ca80:8001:769d::80:c --insecure
GET / HTTP/1.1
Host: web.johncook.co.uk
HTTP/1.1 200 OK …
Web.JohnCook.Co.UK is working, but Web.WatfordJC.Co.UK is having gateway trouble. That is to be expected because the gateway used by nginx for web.watfordjc.co.uk is 127.0.0.1:9000 and I haven't (yet) installed PHP. Also, even if the gateway was working, I haven't copied over /home/www/var/www/ so there aren't any PHP files to process if the processor was installed.
Installing PHP
According to netstat, I am using php5-fpm on vps2. I will use it on vps3 as well then.
sudo apt-get install php5-fpm
sudo so
rsync -avrpPlog --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' thejc@vps2.thejc.me.uk:/etc/php5/fpm/ /etc/php5/fpm/
mkdir -p /home/www/var/www/
mkdir /home/www/var/log/
chown -R www-data:www-data /home/www/
rsync -avrpPlog --progress -e 'ssh -p 8043 -i /home/thejc/.ssh/thejc.vps3' thejc@vps2.thejc.me.uk:/home/www/var/www/ /home/www/var/www/
exit
sudo service php5-fpm restart
sudo service nginx restartWhen testing Web.WatfordJC.Co.UK now, I get a 500 Internal Server Error. A look in /etc/php5/fpm/conf.d/ shows a lot of symbolic links are missing targets: I need to install some php modules.
sudo apt-get install php5-gd php5-mysql php5-readlineI now get a 200 OK response, and a MySQL error 1045 (access denied).
Exporting and Importing MySQL Databases and Users
There are two things that need to be done in order to (hopefully) get these two Web sites functioning. The first is to dump and import the relevant databases. The second is to re-create the users.
The easiest way to re-create specific users is to follow some of the instructions in the article MySQL - Migrate Users from Server to Server. It should be noted I said specific users—I don't want to re-create all the users as it might overwrite existing ones (like root) which is not what I want to do.
mysql -u root -N -p -s > ~/mysql-usersselect Distinct CONCAT('show grants for `', user, '`@`', host, '`;') as query from mysql.user;
quitNow, edit ~/mysql-users and remove all the lines of users we aren't interested in (i.e. all the lines except those for the two database users we want).
With that done and mysql-users saved, we can now get all the grants we need for vps3.
mysql -u root -N -p -s < ~/mysql-users > ~/mysql-grants
scp -P 8043 -i ~/.ssh/thejc.vps2 mysql-grants thejc@vps3.thejc.me.uk:Then, over on vps3, use mysql-grants to create the users:
sed -i 's/$/;/' ~/mysql-grants
mysql -u root -p < ~/mysql-grantsAnother test using gnutls-cli from vps2 and the MySQL error has changed to 1049 (unknown database). We are on the right track.
Now we need to export the two databases:
mysql -u root -pflush tables with read lock;
set global read_only = ON;
exitmysqldump -u root -p thejc_watfordjc > mysql-thejc_watfordjc-2015-05-12.sql
mysqldump -u root -p djlane_djlane > mysql-djlane_djlane-2015-05-12.sql
scp -P 8043 -i ~/.ssh/thejc.vps2 mysql-thejc_watfordjc-2015-05-12.sql thejc@vps3.thejc.me.uk:
scp -P 8043 -i ~/.ssh/thejc.vps2 mysql-djlane_djlane-2015-05-12.sql thejc@vps3.thejc.me.uk:Then, over on vps3:
mysql -u root -pcreate database thejc_watfordjc;
create database djlane_djlane;
flush privileges;
exitmysql -u root -p thejc_watfordjc < ~/mysql-thejc_watfordjc-2015-05-12.sql
mysql -u root -p djlane_djlane < ~/mysql-djlane_djlane-2015-05-12.sqlOne more test from vps2 using gnutls-cli, and /news.php is returned without an error.
Changing IP Addresses
First up, we need to change the backend in varnish on vps3. This is a simple case of editing /etc/varnish/user.vcl, changing fdd7:5938:e2e6:9660::80:c to fdd7:5938:e2e6:6c8d::80:c and changing johncook_vps2_nginx to johncook_vps3_nginx (not forgetting to rename the backend in the fallback block).
A restart of varnish later, and we can now look at the nginx upstream servers.
Edit /etc/nginx/conf.d/upstream-servers.conf and replace :9660: with :6c8d:.
After restarting nginx, we can make the first test change. At this point it is advantageous that Web.JohnCook.UK is on Cloudflare because Cloudflare acts as a reverse proxy. Thus I can make a change to the IPs in Cloudflare and quickly revert without having to worry about DNS TTL waits (propogation).
Changing IP addresses in Cloudflare was a simple matter of copying a new IP to the clipboard, clicking on the value of the A/AAAA record, selecting all the text (triple clicking), pasting, and then clicking outside the table area.
With Cloudflare'd site still functioning, I edited the nsd zone files for the sites that have migrated pointing the A/AAAA records to the new locations.
While the migrated sites are working, there are a much larger number of sites that aren't working as I didn't restore them after restoring vps2 after an attempted upgrade. Those sites aren't a priority at the moment, and in fact most of them are to be merged with my new domains at a later date.
Anyway, at this point I have successfully completed the goal of this article: migrate Web sites from my old VPS to my new one and confirm they are still working. For now I will leave the MySQL databases on vps2 in read-only mode because the only database that may need writing to at a later date is the mail database.
I only add new e-mail addresses once a month on average anyway, but it should be noted this may be an issue at a later time.