A Second VPS

With Ubuntu 16.04 nearing release, I decided it was time for a second VPS.

Background

Every so often I take a look at my current VPS and see whether there is a cheaper (and better) one available from my provider, as well as looking at alternative providers.

It should be noted that I have been using OpenITC as my VPS provider since July 2008, first with XenVZ and more recently with DireVPS. As I wrote in 2015:

As I am mostly satisfied with my current provider, xenvz.co.uk/OpenITC, the obvious question is whether any of DireVPS's packages would be suitable for me and at a price point less than I'm currently paying.
John Cook, VPS Hosting Change in 2015?

They were, and I transitioned to DireVPS. My XenVZ package is still listed in OpenITC as the disk and IP add-ons (SLA compensation) have yet to expire, not that they can be used for anything.

Anyway, for a long time I have been thinking about resiliency which is part of the reason why I started moving my backup DNS from Hurricane Electric to Esgob. Esgob does, however, still have issues with new domains not getting pushed to all of the anycast instances. Combined with Hurricane Electric DNS's failure I wondered if I could get a second VPS and completely manage my own DNS.

I have also been thinking about my Web sites and other services. A specific challenge I want to investigate is a round-robin Web server configuration that supports TLS (including session resumption).

LoveServers as the Provider

As is the case when I look around for what the best deals are, I went to Low End Box (LEB).

After a month or so I decided to give the LoveServers KVM Offer [LowEndBox.com] a try. With the offer, I now have an annually billed VPS costing £15.00 per year (half the cost of my current DireVPS VPS).

KVM Offer

512MB RAM
1 x vCPU
10GB SSD Cached Disk space
250GB Transfer
1Gbps Uplink
1 x IPv4
/64 IPv6
KVM/SolusVM
$15/semi-annual
$25/year

Some things from their TOS that are worth noting:

Your service will be suspended if it receives and or sends any DDoS attack. Service activation or termination after the suspension will be in our sole discretion. LoveServers does not guarantee that any IP address space will be globally accessible due to factors out of our control.

The parties expressly recognize that it is impossible to maintain flawless security, therefore the customer is solely responsible for properly securing their hosting service provided by LoveServers including changing initial passwords and making any security adjustments and patches over time.

LoveServers is not responsible for the cancellation of PayPal subscriptions following the cancellation of a service contract. This is an action that must be carried out by the customer after the service cancellation has been confirmed. Failure to do so may result in additional payments being taken.

LoveServers will suspend services after 2 days of non-payment. After 7 days of non-payment, services will be terminated along with all data, IP addresses and other information. Repeated failure to pay invoices with termination will lead to account termination and refusal of service.

All the following services are banned at LoveServers - running any of the below may result in immediate suspension of services:

Topsites

Anonymizers

Pirated Software/Warez

AutoSurf/PTC/PTS/PPC sites

IP Scanners

Bruteforce Programs/Scripts/Applications

Mail Bombers/Spam Scripts/Mailer Pro

Unsolicited e-mail activities

Link generator scripts for downloading from other file dump sites

Escrow/Bank Debentures

High-Yield Interest Programs (HYIP) or Related Sites

Investment Sites (FOREX, E-Gold Exchange, Second Life/Linden Exchange, Ponzi, MLM/Pyramid Scheme)

Sale of any controlled substance without prior proof of appropriate permit(s)

Prime Banks Programs

Hacker focused sites/archives/programs

Sites promoting illegal activities

Forums and/or websites that distribute or link to warez/pirated/illegal content

Bank Debentures/Bank Debenture Trading Programs

Fraudulent and/or Phishing Sites

Public Proxies / Public VPNs / TOR Exit

Video Encoding / Streaming

Virtual Currency Mining

DDoS-prone applications or booters / flood scripts

Banned services is a rather specific list (some of the terms used almost certainly have the same definition as the Wikipedia articles).

Anonymizer, for example, is defined as an intermediary and privacy shield between a client computer and the rest of the Internet.

Although that definition includes VPNs and TOR, the bullet point near the bottom makes it appear that what they really have a problem with is anonymizers as a service, rather than anonymisers in general.

CJDNS is probably OK between vps4 and vps3/home, but connecting to Hyperboria is probably a no-no unless using Tor as a non-exit node is OK (i.e. darknet nodes OK as long as they don't allow other users on the darknet to go back out to the regular Internet).

Looking at the last few items in general (and the DDoS termination policy) it looks like the main concern is resource usage. Encoding videos and Bitcoin mining is CPU intensive, whilst video streaming and DDoS attacks are bandwidth intensive.

Everything else is either illegal, a type of fraud, computer misuse, or a legally grey area due to dual use (e.g. IP Scanners). When it comes to dual use "hacking" tools, it is probably best to seek permission before using them (like when I asked OpenITC if it is OK to port scan my own VPS from outside their network).

That just leaves the legal unknowns. Linking to pirated content might not be illegal/unlawful, or it might be. Link generator scripts appear to be somewhat akin to a server-side script that can visit a given cyberlocker URL and extract a direct download link for the file at that URL. Assuming deep linking and file lockers isn't the issue, this again looks like a copyright "murky law" issue.

DDoS-prone applications is unclear. Booters, on the other hand, are (from the look of YouTube) things like High Orbit Ion Canon. Flood scripts also launch DDoS attacks at a target. Is "or" used as in "cats or dogs" or as in "felines or cats"? If the former, DDoS-prone must mean juicy targets for DDoS attacks. So, DNS, Web, IRC, pretty much everything.

I therefore believe that this is one of those extremely broad terms/conditions that lets a company terminate any customer's account/service. It doesn't matter if I'm below my bandwidth allowance, I'm hosting a Web site that gets 10 hits a day which is against the Ts and Cs because Web servers are prone to DDoS attacks.

Anyway, the terms of service look OK for the price.

Network Performance

I did use the test file to check download speeds, and experienced exactly the same performance as someone else on Virgin Media VIVID 200 commented on LEB:

wget 'http://lg.loveservers.com/100MB.test' -O /dev/null

Resolving lg.loveservers.com (lg.loveservers.com)... 2a06:8ec0:1:4::dead:beef, 185.55.77.93
Connecting to lg.loveservers.com (lg.loveservers.com)|2a06:8ec0:1:4::dead:beef|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100000000 (95M) [text/plain]
Saving to: ‘/dev/null’

/dev/null                  100%[=======================================>]  95.37M  3.16MB/s   in 32s    

2016-04-18 23:50:43 (2.97 MB/s) - ‘/dev/null’ saved [100000000/100000000]

I also used LoveServers' looking glass to check ping(6) and traceroute(6) from them to vps3 and home. IPv4 looked fine, IPv6 was a bit erratic with ping latency. I expect that's because I'm using a Hurricane Electric (Tunnelbroker.net) tunnel, and wondered if the slow download was also IPv6 related.

wget -4 'http://lg.loveservers.com/100MB.test' -O /dev/null

--2016-04-19 01:09:41--  http://lg.loveservers.com/100MB.test
Resolving lg.loveservers.com (lg.loveservers.com)... 185.55.77.93
Connecting to lg.loveservers.com (lg.loveservers.com)|185.55.77.93|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100000000 (95M) [text/plain]
Saving to: ‘/dev/null’

/dev/null                  100%[=======================================>]  95.37M  21.1MB/s   in 4.9s   

2016-04-19 01:09:46 (19.5 MB/s) - ‘/dev/null’ saved [100000000/100000000]

wget -6 'http://lg.loveservers.com/100MB.test' -O /dev/null

--2016-04-19 01:09:59--  http://lg.loveservers.com/100MB.test
Resolving lg.loveservers.com (lg.loveservers.com)... 2a06:8ec0:1:4::dead:beef
Connecting to lg.loveservers.com (lg.loveservers.com)|2a06:8ec0:1:4::dead:beef|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100000000 (95M) [text/plain]
Saving to: ‘/dev/null’

/dev/null                  100%[=======================================>]  95.37M  4.34MB/s   in 30s    

2016-04-19 01:10:29 (3.23 MB/s) - ‘/dev/null’ saved [100000000/100000000]

Yep, native IPv4 gets 168—180 Mb/s whilst 6in4 IPv6 from HE gets 25–35 Mb/s (a sample size of two for each in a single 15 minute window).

Ordering, Registering, Purchasing, and Logging In

Ordering and registering was rather simple (only snag: my iNum was deemed an invalid phone number) and I soon received 6 e-mails: Customer Invoice, Order Confirmation, Welcome, New Server Information, Credit Card Payment Confirmation, Your LoveServers ltd receipt [#xxxx-xxxx].

The Welcome e-mail says how to login to the client area, and the New Server Information e-mail contained everything else I needed.

The SolusVM login details was my second point of call as I saw very few options in the client area. From here all the typical KVM options are available including reinstalling using their image and booting from an ISO of my own. I am just going to go with LoveServer's image of Ubuntu 14.04 LTS I chose during ordering rather than install from fresh. I changed the timezone to UTC.

This new VPS has been given the unimaginative name vps4. Over on vps3 I added the DNS A/AAAA/SPF records for vps4 using the IPv4 and IPv6 addresses vps4 got allocated.

Although I'm not currently planning on using vps4 for outgoing e-mail, I wanted to double-check there wouldn't be any future issues with DNSBLs. I pasted the IPv4 address into 2 DNSBL checkers/lookup tools and hit the submit buttons. I was glad to see that the IP address is not listed in a single DNSBL.

I then logged in to vps4 as root over SSH.

date

Tue Apr 19 00:23:54 UTC 2016

Quick Configuration

There are several quick things that I need to do to my new VPS to secure it:

Download and apply updates.
Add a new user.
Configure SSH.
Login as new user over SSH.
Secure SSH.
Disable root password login everywhere.

Download and apply updates

After opening /etc/apt/sources.list in nano and changing all of the URLs from http://us.… to http://gb.… I saved and exited nano.

apt-get update && apt-get upgrade
reboot
apt-get update && apt-get dist-upgrade

…
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
…

Open /etc/default/grub and change GRUB_HIDDEN_TIMEOUT from 0 to 10, and GRUB_TIMEOUT from 10 to 0.

update-grub
dpkg-reconfigure tzdata

Choose None of the above, and then choose UTC.

reboot
apt-get autoclean && apt-get autoremove && apt-get update && apt-get dist-upgrade

Add a new user

adduser thejc
adduser thejc sudo

Configure SSH

nano /etc/ssh/sshd_config

Port 8043
LogLevel	VERBOSE
AuthorizedKeysFile	/etc/ssh/%u/authorized_keys

service ssh reload
mkdir /etc/ssh/thejc
chown thejc:root /etc/ssh/thejc
chmod 700 /etc/ssh/thejc

From VPS3, copy my authorized_keys file to vps4.

scp -P 8043 /etc/ssh/thejc/authorized_keys thejc@185.133.75.14:/etc/ssh/thejc/authorized_keys

Back on VPS4, create a symbolic link from my home folder to my ssh folder, and then disable SSH password authentication.

ln -s /etc/ssh/thejc /home/thejc/.ssh
nano /etc/ssh/sshd_config

PasswordAuthentication no

service ssh reload

Login as new user over SSH

At this point, check that everything works. On my laptop:

nano ~/.ssh/config

Host vps4
	IdentityFile ~/.ssh/John-Alienware-JC.key
	Port 8043
Host vps4.johncook.co.uk
	IdentityFile ~/.ssh/John-Alienware-JC.key
	Port 8043

ssh vps4

ssh: Could not resolve hostname vps4.johncook.co.uk: Name or service not known

After all this time I am still waiting for Esgob to make an AXFR.

ssh 185.133.75.14 -p 8043 -i ~/.ssh/John-Alienware-JC.key
sudo su root
exit
exit

Secure SSH

nano /etc/ssh/sshd_config

ListenAddress 2a06:8ec0:1:ec::2da9
ListenAddress 185.133.75.14
PermitRootLogin without-password
AllowUsers thejc

service ssh reload

From laptop, connect to vps4 again and check I can still sudo su root. Disconnect again.

Disable root password login everywhere

Back in the terminal where I'm SSH'd to vps4 as root:

passwd -l root

passwd: password expiry information changed.

passwd -S root

root L 05/06/2014 0 99999 7 -1

The L means that the password for the account is "locked". In other words, the password is set to an invalid value and using it (e.g. to login) is impossible.

exit
ssh vps4
sudo apt-get update && sudo apt-get dist-upgrade
date

Tue Apr 19 01:49:45 UTC 2016

Resource Usage

df -H

Filesystem      Size  Used Avail Use% Mounted on
udev            246M  4.1k  246M   1% /dev
tmpfs            52M  349k   52M   1% /run
/dev/vda1       9.5G  1.7G  7.4G  19% /
none            4.1k     0  4.1k   0% /sys/fs/cgroup
none            5.3M     0  5.3M   0% /run/lock
none            257M     0  257M   0% /run/shm
none            105M     0  105M   0% /run/user

free -m

             total       used       free     shared    buffers     cached
Mem:           489        355        134          0         15        299
-/+ buffers/cache:         40        449
Swap:         1023          0       1023

top

top - 01:59:52 up  1:00,  1 user,  load average: 0.00, 0.01, 0.05
Tasks:  64 total,   1 running,  63 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    501736 total,   364224 used,   137512 free,    15980 buffers
KiB Swap:  1048572 total,        0 used,  1048572 free.   306664 cached Mem

Date and Time Synchronisation

Ubuntu 14.04 comes with ntpdate, which only checks for drift in the time when the network interface comes up. On a server that is typically only after a reboot.

I will be installing openntpd using British NTP servers.

sudo apt-get install openntpd
sudo nano /etc/openntpd/ntpd.conf

# Choose servers announced from Debian NTP Pool
#servers 0.debian.pool.ntp.org
#servers 1.debian.pool.ntp.org
#servers 2.debian.pool.ntp.org
#servers 3.debian.pool.ntp.org

servers 0.uk.pool.ntp.org
servers 1.uk.pool.ntp.org
servers 2.uk.pool.ntp.org
servers 3.uk.pool.ntp.org

sudo service openntpd check && sudo service openntpd restart

Configure Hostname and Hosts File

sudo nano /etc/network/interfaces

Modify the dns-nameservers setting:

#   dns-nameservers 8.8.8.8 208.67.222.222
   dns-nameservers 74.82.42.42 8.8.8.8

sudo rm /etc/resolv.conf
sudo ln -s /run/resolvconf/resolv.conf /etc/resolv.conf
sudo reboot

sudo nano /etc/hostname

vps4

sudo hostname -F /etc/hostname
sudo nano /etc/hosts

# Generated by SolusVM
127.0.0.1               localhost
185.133.75.14           vps4.johncook.co.uk vps4
2a06:8ec0:1:ec::2da9    vps4.johncook.co.uk vps4

# The following lines are desirable for IPv6 capable hosts
::1                     ip6-localhost ip6-loopback
fe00::0                 ip6-localnet
ff00::0                 ip6-mcastprefix
ff02::1                 ip6-allnodes
ff02::2                 ip6-allrouters

Network Issues

LoveServers are having recurrent network issues at the moment.

VPS4 ping graph for 20th April 2016 showing 100% packet loss for an hour.

To quote from an e-mail I received from LoveServers in response to that outage:

It appears that the outage was caused by our datacenter disabling our uplink port due to a port security bug on their cisco switch upstream from us.

As this isn't the first time this has happened, discussions will now begin in regards of staying at this facility or not.

As this won't be my primary VPS—I haven't worked out how I'm going to utilise both of my VPSs yet—the outage wasn't a major issue. It was nice to see that I get e-mail notifications of known issues as I do with OpenITC.

Although LoveServers are young, going by OpenITC and XenVZ's WHOIS records I got my first VPS with OpenITC/XenVZ.co.uk when they were younger than LoveServers are now (7 months old versus 11 months old). I can't remember exactly, but I do recall at some point I was warned that the server containing my VPS was being physically moved to a different data centre.

Found the e-mail:

Our supplier has scheduled a move of our server openvz02.xenvz.co.uk to their new facility. The network capacity is considerably enhanced over the current site we are in.

VPS Node openvz02.xenvz.co.uk will be moving on 10/08/2009. The move will occur during a window from 22:00 on 10/08/2009 and 05:00 on 11/08/2009. All VPS and the server will be powered down, packaged and transported to the new facility.

During the move our supplier takes responsibility for the safety of our equipment but our insurance does not cover data loss or consequential losses – so we recommend you take backups of your data as a precaution. At their new site our server will be re-racked into a pre-prepared environment and checks made by their technicians that connectivity is restored.

Date: 10/08/2009
Window: 22:00 for 7 hours
Duration: < 2 hours

I have no problem with a server move as long as I am given advance notice and the downtime isn't very long (< 12–24 hours depending on distance being moved). OpenITC gave me 7 days notice of the server being physically moved. They also gave me 10 days notice when that VPS moved hardware nodes within a data centre.

Monitoring Network Status

It is a bit hard for someone to determine if packet loss on vps4 is caused by me rebooting the VPS or an issue with an upstream provider, as (unlike for vps3) I am not yet tweeting about reboots and stuff.

Live Think Broadband ping graphs for my servers are, however, available at the following links:

Server Location

UK Population Density in 2011. Image licensed CC by-SA 3.0, Wikimedia Commons. Contains OS data © Crown copyright and database right

Part of the reason why I went with LoveServers, however, was because the VPS would be located in Manchester. Manchester is the approximate centre of the UK in terms of Internet 'backbone' connectivity and population density, as can be seen in this picture from Wikimedia.

In terms of latency (rather than bandwidth usage) geography has a part to play. A lot of people live in South-East England (including myself) so Maidenhead or somewhere else close to (or within) London makes sense for a first location.

London is also a good place for UK-Wide and worldwide connectivity due to so many peers at the LINX Internet exchange.

Anyway, as you can see in the population map, after London you have Manchester. After Manchester you have Glasgow/Edinburgh, and then after that you have Birmingham followed by Newcastle upon Tyne, Cardiff, and Belfast.

Wikipedia's list of IXs lists 9 in the UK. 3 of those are in London (LINX, LONAP, NetIX), 2 in Manchester (IXManchester, mcix), and 1 in each of Brighton (IX Brighton), Cardiff (IXCardiff), Edinburgh (IXScotland), and Leeds (IXLeeds).

By going by population and connectivity (closeness to well-peered IXs) it is possible to work out where a failover server should be located. Almost all ISPs peer at LINX, to the point some traceroutes from Newcastle to Manchester will travel via LINX.

If there is a routing issue at an ISP (which invariably in the UK is an issue with a peer at LINX), then Manchester is a good choice of secondary location. Some ISPs (e.g. Virgin Media) peer at IXManchester meaning if the VPS provider's IP range is announced by a peer at IXManchester traffic between ISP users and that VPS will not travel via LINX… if they peer with each other and traffic is routed via IXManchester.

traceroute vps4.johncook.co.uk

traceroute to vps4.johncook.co.uk (185.133.75.14), 30 hops max, 60 byte packets
 1  * * *
 2  watf-core-2a-xe-022-0.network.virginmedia.net (80.3.2.213)  11.642 ms  12.576 ms  12.688 ms
 3  brnt-bb-1a-ae7-0.network.virginmedia.net (62.253.175.90)  26.762 ms  26.768 ms  19.553 ms
 4  tclo-ic-2-ae0-0.network.virginmedia.net (212.250.14.202)  19.753 ms  19.581 ms  19.722 ms
 5  195.66.238.78 (195.66.238.78)  24.994 ms  25.936 ms  26.023 ms
 6  37.220.95.86.srvlist.ukfast.net (37.220.95.86)  26.089 ms  23.088 ms  22.325 ms
 7  . (185.55.77.92)  18.181 ms  17.918 ms  18.841 ms
 8  185.133.75.14 (185.133.75.14)  25.371 ms  25.436 ms  20.283 ms

Well, that is the theory. The reason that traceroute went through LINX (195.66.238.78) could either be because LoveServers aren't peering with Virgin Media at IXManchester, or because going via LINX was the preferred/shortest route at the time.

According to Hurricane Electric, LoveServers' IP range is announced by UKFast (AS34934). The only IPv4 prefix announced by LoveServers themselves is 185.145.44.0/24 and they only peer with M247 Limited (AS9009).

Since vps4's IP address isn't in the single (IPv4) network announced by LoveServers, the only route to my VPS must go through UKFast. According to loveservers on LowEndTalk:

UKFast recently changed their peering at IXManchester so that they only privately peer with a small number of ISPs. We've been trying to persuade them to be a little less picky.
loveservers, LowEndTalk comment on 26th February 2016

According to LINX, UKFast have an open peering policy (an inclination to peer with anyone) and Virgin Media have a restrictive peering policy (an inclination not to peer with any more entities). Virgin Media have a 10G port on router 195.66.244.33 at IXManchester and UKFast have a 1G port on router 195.66.244.25 (IPv6: 2001:7f8:4:2::8876:1) at IXManchester.

Something worth noting is that vps4 has so far always had quicker ping response (round trip) times than vps3, even though the speed of light should mean that isn't the case. From here to Telehouse North is approximately 25 miles. Telehouse North to Maidenhead is approximately 40 miles. Telehouse North to Manchester is approximately 225 miles.

Therefore an as-the-crow-flies distance (one direction) from home to vps3 via Docklands is approximately 65 miles and from home to vps4 via Docklands approximately 250 miles. It should take data longer to travel 500 miles than it does to travel 130 miles, but that isn't the case here.

The Maidenhead to London to Manchester fibre must be pretty good. 530 miles round trip in less than 8 milliseconds. Assuming data travels at 60% of the speed of light in fibre, it can travel 894 miles in 8 milliseconds. Put another way, 4.74 milliseconds is currently the fastest data can travel 530 miles, and that 530 miles assumes it is a straight line from vps3 to Docklands and another straight line from Docklands to vps4 (rather unlikely).

Given VPS4 has better round trip times than VPS3 for someone only 21 miles as-the-crow-flies from Telehouse North, traffic from Newcastle (or anywhere else in the UK) to my VPS in Manchester will probably have the same latency as it would going to my VPS in Maidenhead if it goes via LINX.

Therefore although LINX may be a problem at times, until UKFast peers with more UK ISPs at IXManchester I can treat vps3 and vps4 as if they are at the same location if I decide to do anything like geographical DNS.

Firewall (iptables)

On my Debian-based systems my files for iptables are stored in /etc/network/ and are typically called iptables.bak or iptables.save For ip6tables the files are similarly called ip6tables.bak or ip6tables.save

IPv4

sudo nano /etc/network/iptables.save

I have commented some lines #'# to indicate they have been copied from vps3 where they are active, to differentiate from rules that are also disabled on vps3.

*filter
:INPUT DROP
:FORWARD ACCEPT
:OUTPUT ACCEPT
:in-new -

### Permit localhost traffic ###
-A INPUT -s 127.0.0.1/8 -d 127.0.0.1/8 -j ACCEPT


### ICMP ###
# Ignore fragmented ICMP packets
-A INPUT -p icmp --fragment -j DROP

# Permit ICMP echo requests to primary IP
-A INPUT -d 185.133.75.14 -p icmp --icmp-type echo-request -j ACCEPT
# Permit ICMP echo replies from primary IP
-A OUTPUT -s 185.133.75.14 -p icmp --icmp-type echo-reply -j ACCEPT

# Permit ICMP echo requests from primary IP
-A OUTPUT -s 185.133.75.14 -p icmp --icmp-type echo-request -j ACCEPT
# Permit ICMP echo replies to primary IP
-A INPUT -d 185.133.75.14 -p icmp --icmp-type echo-reply -j ACCEPT

# Permit ICMP Path MTU Discovery to/from all IPs
-A OUTPUT -p icmp --icmp-type fragmentation-needed -j ACCEPT
-A INPUT -p icmp --icmp-type fragmentation-needed -j ACCEPT

# Permit ICMP TTL exceeded to/from all IPs
-A OUTPUT -p icmp --icmp-type time-exceeded -j ACCEPT
-A INPUT -p icmp --icmp-type time-exceeded -j ACCEPT

# Permit UDP traceroute rejections
-A INPUT -p udp --dport 33434:33523 -j REJECT
-A OUTPUT -p icmp --icmp-type port-unreachable -j ACCEPT

# Ignore all other ICMP requests/replies
-A OUTPUT -p icmp -j DROP
-A INPUT -p icmp -j DROP


### All non-ICMP packets ###
# Allow outbound packets for new and existing/related connections
-A OUTPUT -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT
# Allow inbound packets belonging to an established or related connection
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Silently drop inbound Packets that are out-of-sequence
-A INPUT -m conntrack --ctstate INVALID -j DROP
# New inbound connections unknown to the kernel are handled in a separate chain
-A INPUT -m conntrack --ctstate NEW -j in-new


### SSH ###
# Permit SSH
-A in-new -p tcp --dport 8043 --syn -j ACCEPT

### DNS ###
# Permit DNS requests to primary IP
#'#-A in-new -d 185.133.75.14 -p tcp --dport 53 -j ACCEPT
#'#-A in-new -d 185.133.75.14 -p udp --dport 53 -j ACCEPT

### CJDNS ###
#'#-A in-new -d 185.133.75.14 -p udp --dport 27177 -j ACCEPT

### SMTP ###
# Permit SMTP connections for inbound e-mail
#'#-A in-new -d 185.133.75.14 -p tcp --dport 25 -j ACCEPT

### Web ###
# Permit HTTP(S) connections to primary IP
#'#-A in-new -d 185.133.75.14 -p tcp -m multi --dports 80,443 -j ACCEPT


COMMIT

*nat
:PREROUTING ACCEPT
:INPUT ACCEPT
:OUTPUT ACCEPT
:POSTROUTING ACCEPT
COMMIT

sudo iptables-restore < /etc/network/iptables.save

Check I can still SSH in over IPv4 from my laptop:

ssh -4 vps4
exit

IPv6

sudo nano /etc/network/ip6tables.save

I have commented some lines #'# to indicate they have been copied from vps3 where they are active, to differentiate from rules that are also disabled on vps3.

*filter
:INPUT DROP
:FORWARD ACCEPT
:OUTPUT ACCEPT
:in-new -
:permitted-dns -
:permitted-mysql -


### Permit loopback traffic ###
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

# Allow some ICMP traffic
#'#-A INPUT -i ula-net -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
#'#-A INPUT -i ula-net -p ipv6-icmp -j ACCEPT
-A INPUT -i eth0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p ipv6-icmp -j ACCEPT

# Needed for tunnel
-A INPUT -p ipv6-nonxt -j ACCEPT
-A INPUT -p udp -m udp --dport 32768:61000 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 32768:61000 ! --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT

#'#-A OUTPUT -o ula-net -d fdd7:5938:e2e6::/48 -j ACCEPT
#'#-A INPUT -i eth0 -d  fc00::/7 -j REJECT --reject-with icmp6-no-route
#-A OUTPUT -o eth0 -d  fc00::/7 -j LOG --log-prefix "eth0 block ula"
#'#-A OUTPUT -o eth0 -d  fc00::/7 -j REJECT --reject-with icmp6-no-route

# packets belonging to an establish connection or related to one can pass
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# packets that are out-of-sequence are silently dropped
-A INPUT -m conntrack --ctstate INVALID -j DROP
# new connections unknown to the kernel are handled in a separate chain
-A INPUT -m conntrack --ctstate NEW -j in-new

### Pass SYN packets for SSH ###
-A in-new -d 2a06:8ec0:1:ec::2da9 -p tcp --dport 8043 --syn -j ACCEPT

### ULA Network ###
# Home LAN
#'#-A INPUT -i tun0 -s fc1f:a237:8377:8d3b:db9b:4221:471:f46a/128 -d fc89:55f4:d87:ed52:a66:da7d:e906:6544/128 -j ACCEPT
#'#-A FORWARD -i tun0 -o ula-net -s fc1f:a237:8377:8d3b:db9b:4221:471:f46a/128 -d fc89:55f4:d87:ed52:a66:da7d:e906:6544/128 -j ACCEPT
#'#-A INPUT -i ula-net -s fdd7:5938:e2e6::/48 -d fdd7:5938:e2e6:6c8d::/64 -j ACCEPT
#'#-A OUTPUT -o ula-net -s fdd7:5938:e2e6:6c8d::/64 -d fdd7:5938:e2e6:1::/64 -j ACCEPT
# VPS2
#'#-A INPUT -i ula-net -s fdd7:5938:e2e6:9660::/64 -d fdd7:5938:e2e6:6c8d::/64 -j ACCEPT
#'#-A OUTPUT -o ula-net -s fdd7:5938:e2e6:6c8d::/64 -d fdd7:5938:e2e6:9660::/64 -j ACCEPT
# Endpoints
#'#-A INPUT -i ula-net -s fdd7:5938:e2e6:3::/64 -d fdd7:5938:e2e6::/48 -j ACCEPT
#'#-A OUTPUT -o ula-net -s fdd7:5938:e2e6::/48 -d fdd7:5938:e2e6:3::/64 -j ACCEPT

### ICMP ###
-A INPUT -p icmpv6 -j ACCEPT -m limit --limit 30/min
-A FORWARD -p icmpv6 -j ACCEPT


### DNS ###
#'#-A in-new -d 2a03:ca80:8001:769d::5 -p udp --dport 53 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::5 -p tcp --dport 53 -j ACCEPT

### ULA Tunneling Restrictions ###
# ULA traffic from Home LAN via Hurricane Electric
#'#-A in-new -i eth0 -s 2001:470:1f09:1aab::2 -d 2a03:ca80:8001:769d::2 -j ACCEPT
#'#-A FORWARD -i eth0 -s 2001:470:1f09:1aab::2 -d 2a03:ca80:8001:769d::2 -j ACCEPT
# ULA traffic from Home LAN via CJDNS
#'#-A in-new -i tun0 -s fc1f:a237:8377:8d3b:db9b:4221:471:f46a/128 -d fc89:55f4:d87:ed52:a66:da7d:e906:6544/128 -j ACCEPT
#'#-A FORWARD -i tun0 -s fc1f:a237:8377:8d3b:db9b:4221:471:f46a/128 -d fc89:55f4:d87:ed52:a66:da7d:e906:6544/128 -j ACCEPT
# ULA traffic from VPS2 via Hurricane Electric
#'#-A in-new -i eth0 -s 2001:470:1f09:38d::2 -d 2a03:ca80:8001:769d::2 -j ACCEPT
#'#-A FORWARD -i eth0 -s 2001:470:1f09:38d::2 -d 2a03:ca80:8001:769d::2 -j ACCEPT
# ULA traffic from VPS2 via CJDNS
#'#-A in-new -i tun0 -s fc33:e8dd:42c3:9871:ddca:cb6d:1aad:8c19 -d fc89:55f4:d87:ed52:a66:da7d:e906:6544/128 -j ACCEPT
#'#-A FORWARD -i tun0 -s fc33:e8dd:42c3:9871:ddca:cb6d:1aad:8c19 -d fc89:55f4:d87:ed52:a66:da7d:e906:6544/128 -j ACCEPT

### SMTP ###
#'#-A in-new -d fdd7:5938:e2e6:6c8d:7f00:1:c:8891 -p tcp --dport 8891 -j ACCEPT
#'#-A in-new -d fdd7:5938:e2e6:6c8d:7f00:1:c:8893 -p tcp --dport 8893 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::25:4 -p tcp -m multiport --dports 25,443,587,993 -j ACCEPT
### Let's Encrypt ###
#-A in-new -d 2a03:ca80:8001:769d::25:4 -p tcp -m multiport --dports 80 -j ACCEPT

### Web ###
#'#-A in-new -d fdd7:5938:e2e6:6c8d::6081:c -p tcp --dport 6081 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::80:1 -p tcp -m multiport --dports 80,443 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::80:2 -p tcp -m multiport --dports 80,443 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::80:3 -p tcp -m multiport --dports 80,443 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::80:4 -p tcp -m multiport --dports 80,443 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::80:a -p tcp -m multiport --dports 80,443 -j ACCEPT
#'#-A in-new -d 2a03:ca80:8001:769d::80:c -p tcp -m multiport --dports 80,443 -j ACCEPT

COMMIT

sudo ip6tables-restore < /etc/network/ip6tables.save

Check I can still SSH in over IPv6 from my laptop:

ssh -6 vps4
exit

Now to load the rules on boot. Add the following to /etc/network/interfaces before the IPv6 addresses and routes are added (i.e. before the up rules:

   pre-up /sbin/iptables-restore < /etc/network/iptables.save
   pre-up /sbin/ip6tables-restore < /etc/network/ip6tables.save

Upgrade to Ubuntu 16.04

At this point Ubuntu 16.04 was released.

sudo do-release-upgrade -c -d

Checking for a new Ubuntu release
New release '16.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Be Careful

The -c switch checks if an upgrade is available and tells you what that upgrade is without upgrading to it.

The -d switch checks if a development version is available to be upgraded to and then upgrades you to it.

The value of Prompt in /etc/update-manager/release-upgrades determines what upgrades will be available. If Prompt=lts then only upgrades to LTS versions will be performed.

The following Web pages are used for upgrades:

do-release-upgrade when Prompt=normal: meta-release
do-release-upgrade when Prompt=lts: meta-release-lts
do-release-upgrade -d when Prompt=normal: meta-release-development
do-release-upgrade -d when Prompt=lts: meta-release-lts-development
do-release-upgrade -p when Prompt=normal: meta-release-proposed
do-release-upgrade -p when Prompt=lts: meta-release-lts-proposed

sudo do-release-upgrade -d

Checking for a new Ubuntu release
Get:1 Upgrade tool signature [198 B]                                                                   
Get:2 Upgrade tool [1,264 kB]                                                                          
Fetched 1,264 kB in 0s (0 B/s)                                                                         
authenticate 'xenial.tar.gz' against 'xenial.tar.gz.gpg' 
extracting 'xenial.tar.gz'

Reading cache

Checking package manager

Continue running under SSH? 

This session appears to be running under ssh. It is not recommended 
to perform a upgrade over ssh currently because in case of failure it 
is harder to recover. 

If you continue, an additional ssh daemon will be started at port 
'1022'. 
Do you want to continue? 

Continue [yN]

Before continuing, modify iptables and ip6tables rules so that syn packets to port 1022 is let through. Then press y and hit enter.

Continue [yN] y

Starting additional sshd 

To make recovery in case of failure easier, an additional sshd will 
be started on port '1022'. If anything goes wrong with the running 
ssh you can still connect to the additional one. 
If you run a firewall, you may need to temporarily open this port. As 
this is potentially dangerous it's not done automatically. You can 
open the port with e.g.: 
'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT' 

To continue please press [ENTER]

When prompted, install the package maintainer's version of the following:

/etc/openntpd/ntpd.conf
/etc/default/grub

Once the upgrade is complete and the system has rebooted, make the same changes I made earlier to those files, update grub, and reboot.

sudo update-grub
sudo reboot

Comment out the lines in iptables.save and ip6tables.save and run iptables-restore and ip6tables-restore.

sudo apt-get update && sudo apt-get dist-upgrade && sudo apt-get autoremove && sudo apt-get autoclean

Authoritative Name Server

This section is based on my previous article Using NSD as My Master Nameserver and Digital Ocean's How To Use NSD, an Authoritative-Only DNS Server, on Ubuntu 14.04.

I use the latest version of NSD as an authoritative name server.

sudo mkdir /opt/nsd
sudo chown thejc:thejc /opt/nsd
cd /opt
sudo apt-get install subversion
svn checkout http://www.nlnetlabs.nl/svn/nsd/trunk nsd
cd nsd
svn switch ^/tags/$(svn ls ^/tags | egrep ^NSD_.*_REL/$ | tail -n 1)
nice -n 19 ionice -c 3 sudo apt-get install libevent-dev autoconf gcc openssl libssl-dev make bison flex
sudo useradd -r nsd
nice -n 19 ionice -c 3 autoreconf
nice -n 19 ionice -c 3 ./configure
nice -n 19 ionice -c 3 make
nice -n 19 ionice -c 3 sudo make install
sudo nsd-control-setup
sudo ip addr add 2a06:8ec0:1:ec::6/64 dev eth0
sudo nano /etc/network/interfaces

Add the new IPv6 address under the /64 subnet line.

   up ip addr add 2a06:8ec0:1:ec::2da9/64 dev eth0
   up ip addr add 2a06:8ec0:1:ec::6/128 dev eth0

cd /etc/nsd
sudo cp nsd.conf.sample nsd.conf
sudo nano nsd.conf

server:
	server-count: 2
	
	ip-address: 127.0.0.1
	ip-address: 185.133.75.14
	ip-address: 2a06:8ec0:1:ec::6
	
	do-ip4: yes
	
	do-ip6: yes
	
	username: nsd
	
	zonesdir: "/etc/nsd/zones"

	logfile: "/var/log/nsd.log"

# Remote control config section.
remote-control:
        control-enable: yes

        control-interface: 127.0.0.1
        control-interface: ::1

        control-port: 8952

        server-key-file: "/etc/nsd/nsd_server.key"

        server-cert-file: "/etc/nsd/nsd_server.pem"

        control-key-file: "/etc/nsd/nsd_control.key"

        control-cert-file: "/etc/nsd/nsd_control.pem"

pattern:
	name: "myzones"
	
	zonefile: "%s.zone"
	
pattern:
	name: "vps3master"
	
	include-pattern: "myzones"
	
	allow-notify: 149.255.108.141 NOKEY
	
	request-xfr: 149.255.108.141 NOKEY

	allow-axfr-fallback: yes
	
	outgoing-interface: 185.133.75.14


zone:
	name: "thejc.me.uk"
	include-pattern: "henetslaves"
zone:
	name: "johncook.co.uk"
	include-pattern: "esgobslaves-not-henet"
zone:
	name: "watfordjc.co.uk"
	include-pattern: "henetslaves"
zone:
	name: "johncook.uk"
	include-pattern: "esgobslaves"
zone:
	name: "watfordjc.uk"
	include-pattern: "esgobslaves"

sudo mkdir zones
sudo chown nsd:nsd zones
sudo mkdir -p /usr/lib/systemd/system/
sudo nano /usr/lib/systemd/system/nsd.service

[Unit]
Description=Name Server Daemon
After=network.target

[Service]
Type=simple
Restart=always
Environment=CONFFILE=/etc/nsd/nsd.conf
ExecStartPre=/bin/sh -c '/bin/mkdir -p "$(dirname "$(/usr/local/sbin/nsd-checkconf -o pidfile $CONFFILE)")"'
ExecStartPre=/bin/sh -c '/bin/chown "$(/usr/local/sbin/nsd-checkconf -o username $CONFFILE)" "$(dirname "
$(/usr/local/sbin/nsd-checkconf -o pidfile $CONFFILE)")"'
ExecStartPre=/bin/sh -c '/bin/mkdir -p "$(dirname "$(/usr/local/sbin/nsd-checkconf -o database $CONFFILE)")"'
ExecStartPre=/bin/sh -c '/bin/chown "$(/usr/local/sbin/nsd-checkconf -o username $CONFFILE)" "$(dirname "
$(/usr/local/sbin/nsd-checkconf -o database $CONFFILE)")"'
ExecStart=/usr/local/sbin/nsd -d -c $CONFFILE
ExecReload=/usr/local/sbin/nsd-control reload

[Install]
WantedBy=multi-user.target

sudo systemctl enable nsd
sudo nsd-checkconf /etc/nsd/nsd.conf && sudo service nsd start

TSIG Support

To use TSIG to make sure no MITM is messing with the AXFR requests/responses, a shared key needs to be generated. I'll do this on my laptop as it has a lot of entropy.

dd if=/dev/random of=/dev/stdout count=1 bs=32 | base64

Zk35/IoZ3kTbX1goBn2fVlWZtYT/m2kIUj1lq3Ev2/Y=
1+0 records in
1+0 records out
32 bytes (32 B) copied, 0.000120478 s, 266 kB/s

Copy the random string and use it in a key: on both vps3 and vps4:

sudo nano /etc/nsd/nsd.conf

key:
	name: "vps3key"
	algorithm: hmac-sha256
	secret: "jTPv9PcCo4tkgJ/MAC0990UH"

Now just change NOKEY to vps3key on vps4 for allow-notify and request-xfrin pattern vps3master, and on vps3 in pattern myzones.

Then, on both servers, check the configuration is OK and restart nsd:

sudo nsd-checkconf /etc/nsd/nsd.conf && sudo service nsd restart

Wait until the next SOA check to ensure everything is working correctly, or on vps4:

sudo nsd-control force_transfer

Finishing Up

Things on my new VPS aren't anywhere near setup yet and there is still a lot to do.

One thing that can get annoying, however, is an unexpected fsck during a server reboot.

On laptop, install ssvnc:

sudo apt-get install tightvnc-java

Open Remote Desktop Viewer from the Applications/Utilities menu and configure it using the credentials in SolusVM:

Protocol: VNC
Host: ip:port

Click connect and then enter password from SolusVM.

In a vps4 terminal:

sudo touch /forcefsck
sudo shutdown -r now