New Domain Names - Part 2:
Moving URLs

In part 2 of this series of articles I look at what URLs need moving to the new domains.

Introduction

Before moving my domains, I first need a list of URLs that currently exist. Before doing that, however, I need to create TLS certificates for the new domains.

New TLS Keys and Certificates

To start, I logged in to StartSSL with my S/MIME (client) certificate, and then went to validations in order to validate my new domains.

At this point I hit a snag: domain validation requires the receipt of e-mails to required e-mail addresses and my new domains do not have MX records (although they do have SPF and SPF(TXT) records so others cannot forge FROM headers for spam purposes.

TinyDNS, MX Records, and Domain Validation

My current set-up is to use my VPS as the authoritative nameserver for my domains, with Hurricane Electric for slave DNS server(s). Thus, the zone for WatfordJC.UK is as follows after making the needed changes:

Zwatfordjc.uk:ns5.thejc.me.uk:hostmaster.johncook.co.uk:2015011301:10800:900:1209600:60
.watfordjc.uk::ns5.thejc.me.uk:172800
.watfordjc.uk::ns7.thejc.me.uk:172800
.watfordjc.uk::ns5.he.net:172800

'watfordjc.uk:v=spf1 -all:7200
:watfordjc.uk:99:\013v=spf1 -all:7200
'_domainkey.watfordjc.uk:o=-; t=n;:7200
'_adsp._domainkey.watfordjc.uk:dkim=discardable; t=s:600
'_dmarc.watfordjc.uk:v=DMARC1; p=reject; pct=100; adkim=s; aspf=s; rua=mailto\072postmaster@thejc.me.uk;:60

@watfordjc.uk::mail3.thejc.me.uk:1

+watfordjc.uk:149.255.97.82:3600
3watfordjc.uk:200104701f09038d000000000080000c:60

+web.watfordjc.uk:149.255.97.82:3600
3web.watfordjc.uk:200104701f09038d000000000080000c:60
'web.watfordjc.uk:v=spf1 -all:7200
:web.watfordjc.uk:99:\013v=spf1 -all:7200

In order to receive incoming e-mail for the postmaster/hostmaster in order to validate ownership of the domain with StartSSL, I needed to add MX records. While I was at it, I also added the A/AAAA/SPF/SPF(TXT) records for web.watfordjc.uk.

After re-validating the domain with Hurricane Electric DNS, the changes got pulled by HE after about 10 or so minutes.

At that point it was a simple case of adding the domains to my mail database, creating aliases for mailboxes that are required by the RFCs, and sending a test message from Gmail to the postmaster at the new domains and checking everything was working. It was.

Domain Validation With StartSSL

Now that e-mail was confirmed to be working, I validated my domains with StartSSL, and then generated a SHA-256 certificate signing request (CSR) using a 4096-bit RSA key:

sudo su
cd /etc/ssl/
mkdir -p web.watfordjc.uk/201501/
cd watfordjc.uk/201501/
openssl genrsa -out web_watfordjc_uk.key 4096 -sha256
openssl req -new -key web_watfordjc_uk.key -out web_watfordjc_uk.csr -sha256

Certificate Creation

With web_watfordjc_uk.csr open, I copied and pasted the content to StartSSL (after skipping the key generation page), selected the domain watfordjc.uk, and then created a certificate for web.watfordjc.uk.

StartSSL then told me it has successfully created a certificate, which I then copied and pasted into web_watfordjc_uk.crt

At this point I had everything needed, but needed two more files: one chaining the certificate with the intermediate certificate, and a second with the certificate and the key. Following my naming convention for such files, these were created with the following commands (using the same root shell above) and then changing the permissions to more restrictive ones:

cat web_watfordjc_uk.crt sub.class1.server.sha2.ca.pem > web_watfordjc_uk.chained-no-root
cat web_watfordjc_uk.crt web_watfordjc_uk.key > web_watfordjc_uk.pem
chmod 400 web_*
exit

In this entire section where I have used the plural "domains", I have been referring to both WatfordJC.UK and JohnCook.UK and/or Web.WatfordJC.UK and Web.JohnCook.UK

Spidering Old Domains

The domains for which old content will be moving to the new domains are:

  • TheJC.me.uk
  • www.thejc.me.uk
  • blogs.thejc.me.uk
  • watfordjc.com
  • www.watfordjc.com
  • watfordjc.co.uk
  • www.watfordjc.co.uk
  • web.watfordjc.com
  • johncook.co.uk
  • www.johncook.co.uk
  • web.johncook.co.uk

Google's Index of WatfordJC.com

The first obvious start is to check what URLs still exist for watfordjc.com (and subdomains). A Google search reveals the following:

  • https://watfordjc.com/
  • https://watfordjc.com/news.php
  • https://watfordjc.com/search.php
  • https://watfordjc.com/articles.php?article_id=69
  • https://www.watfordjc.com/articles.php?article_id=73
  • https://watfordjc.com/articles.php?article_id=52
  • https://watfordjc.com/articles.php?article_id=45
  • https://watfordjc.com/articles.php?article_id=24
  • https://watfordjc.com/articles.php?article_id=28
  • https://watfordjc.com/articles.php?article_id=18
  • https://watfordjc.com/articles.php?article_id=25
  • https://watfordjc.com/articles.php?article_id=40
  • https://watfordjc.com/articles.php?article_id=57
  • https://watfordjc.com/articles.php?article_id=16
  • https://watfordjc.com/articles.php?article_id=43
  • https://watfordjc.com/articles.php?article_id=55
  • https://watfordjc.com/articles.php?article_id=47
  • https://watfordjc.com/articles.php?article_id=11
  • https://watfordjc.com/articles.php?article_id=37
  • https://watfordjc.com/articles.php?article_id=33
  • https://watfordjc.com/articles.php?article_id=53
  • https://watfordjc.com/articles.php?article_id=14
  • https://watfordjc.com/articles.php?article_id=27
  • https://watfordjc.com/articles.php?article_id=20
  • https://watfordjc.com/articles.php?article_id=48
  • https://www.watfordjc.com/articles.php?cat_id=7
  • https://watfordjc.com/articles.php?article_id=12
  • https://watfordjc.com/articles.php?article_id=54
  • https://www.watfordjc.com/articles.php?article_id=73&rowstart=6
  • https://www.watfordjc.com/articles.php?article_id=73&rowstart=2
  • https://www.watfordjc.com/weblinks.php?cat_id=2&weblink_id=1
  • https://www.watfordjc.com/articles.php?article_id=73&rowstart=5
  • https://watfordjc.com/articles.php?article_id=34
  • https://watfordjc.com/articles.php?article_id=17
  • https://watfordjc.com/articles.php?article_id=64
  • https://watfordjc.com/articles.php?article_id=51
  • https://watfordjc.com/articles.php?article_id=38
  • https://www.watfordjc.com/articles.php?article_id=29
  • https://www.watfordjc.com/articles.php?cat_id=8
  • https://www.watfordjc.com/articles.php?article_id=5
  • https://www.watfordjc.com/articles.php?article_id=46
  • https://watfordjc.com/articles.php?cat_id=2&rowstart=0
  • https://www.watfordjc.com/articles.php?article_id=49
  • https://watfordjc.com/articles.php?cat_id=2&rostart=30
  • https://watfordjc.com/articles.php?article_id=69&rowstart=0
  • https://www.watfordjc.com/articles.php?article_id=73&rowstart=0
  • https://www.watfordjc.com/articles.php?article_id=69&rowstart=1

As can be seen, a lot of articles are pages are still in the Google index from the domain (www.)watfordjc.com, which I expect is mainly because the redirects in place to web.watfordjc.co.uk are only temporary redirects.

The next logical domain to have a look at is watfordjc.co.uk, as I am going to likely have some overlap with watfordjc.com and some pages being indexed under both domains (and/or sub-domains).

Google's Index of WatfordJC.co.uk

  • https://web.watfordjc.co.uk/
  • https://web.watfordjc.co.uk/news_cats.php
  • https://web.watfordjc.co.uk/search.php
  • https://web.watfordjc.co.uk/articles.php
  • https://web.watfordjc.co.uk/weblinks.php
  • https://web.watfordjc.co.uk/login.php
  • https://web.watfordjc.co.uk/photogallery.php
  • https://web.watfordjc.co.uk/lostpassword.php
  • https://web.watfordjc.co.uk/downloads.php
  • https://web.watfordjc.co.uk/news.php?readmore=6
  • https://web.watfordjc.co.uk/articles.php?article_id=66
  • https://web.watfordjc.co.uk/articles.php?article_id=59
  • https://web.watfordjc.co.uk/articles.php?article_id=73
  • https://web.watfordjc.co.uk/articles.php?cat_id=2
  • https://web.watfordjc.co.uk/news.php?readmore=3
  • https://web.watfordjc.co.uk/news.php?readmore=8
  • https://web.watfordjc.co.uk/articles.php?article_id=69
  • https://web.watfordjc.co.uk/articles.php?article_id=60
  • https://web.watfordjc.co.uk/news.php?readmore=5
  • https://web.watfordjc.co.uk/articles.php?article_id=68
  • https://web.watfordjc.co.uk/articles.php?article_id=22
  • https://web.watfordjc.co.uk/articles.php?article_id=30
  • https://web.watfordjc.co.uk/articles.php?article_id=33
  • https://web.watfordjc.co.uk/articles.php?article_id=65
  • https://web.watfordjc.co.uk/articles.php?article_id=1
  • https://web.watfordjc.co.uk/articles.php?article_id=10
  • https://web.watfordjc.co.uk/articles.php?cat_id=3
  • https://web.watfordjc.co.uk/articles.php?article_id=36
  • https://web.watfordjc.co.uk/weblinks.php?cat_id=1
  • https://web.watfordjc.co.uk/articles.php?article_id=21
  • https://web.watfordjc.co.uk/articles.php?article_id=15
  • https://web.watfordjc.co.uk/articles.php?article_id=20
  • https://web.watfordjc.co.uk/articles.php?article_id=56
  • https://web.watfordjc.co.uk/weblinks.php?cat_id=3
  • https://web.watfordjc.co.uk/articles.php?article_id=31
  • https://web.watfordjc.co.uk/articles.php?cat_id=9
  • https://web.watfordjc.co.uk/articles.php?article_id=73&rowstart=3
  • https://web.watfordjc.co.uk/articles.php?article_id=73&rowstart=2
  • https://web.watfordjc.co.uk/articles.php?article_id=73&rowstart=6
  • https://web.watfordjc.co.uk/weblinks.php?cat_id=2&weblink_id=1
  • https://web.watfordjc.co.uk/articles.php?article_id=68&rowstart=1
  • https://web.watfordjc.co.uk/articles.php?article_id=73&rowstart=4
  • https://web.watfordjc.co.uk/gpg_keys/8FCF5BB.asc
  • https://web.watfordjc.co.uk/gpg_keys/C5009D27.asc
  • https://web.watfordjc.co.uk/gpg_keys/43F159F0.asc
  • https://web.watfordjc.co.uk/articles.php?article_id=4
  • https://web.watfordjc.co.uk/articles.php?article_id=29
  • https://web.watfordjc.co.uk/articles.php?article_id=53
  • https://web.watfordjc.co.uk/articles.php?cat_id=1
  • https://web.watfordjc.co.uk/news.php?readmore=2
  • https://web.watfordjc.co.uk/articles.php?article_id=14
  • https://web.watfordjc.co.uk/articles.php?cat_id=5
  • https://web.watfordjc.co.uk/articles.php?cat_id=7
  • https://web.watfordjc.co.uk/news_cats.php?cat_id=3
  • https://web.watfordjc.co.uk/articles.php?article_id=34
  • https://web.watfordjc.co.uk/news.php?readmore=12
  • https://web.watfordjc.co.uk/articles.php?article_id=55
  • https://web.watfordjc.co.uk/articles.php?article_id=5
  • https://web.watfordjc.co.uk/weblinks.php?cat_id=2
  • https://web.watfordjc.co.uk/articles.php?article_id=11
  • https://web.watfordjc.co.uk/articles.php?article_id=46
  • https://web.watfordjc.co.uk/articles.php?article_id=8
  • https://web.watfordjc.co.uk/news.php?readmore=1
  • https://web.watfordjc.co.uk/articles.php?article_id=9
  • https://web.watfordjc.co.uk/articles.php?article_id=7
  • https://web.watfordjc.co.uk/articles.php?article_id=28
  • https://web.watfordjc.co.uk/photogallery.php?photo_id=1
  • https://web.watfordjc.co.uk/articles.php?article_id=18
  • https://web.watfordjc.co.uk/articles.php?cat_id=10
  • https://web.watfordjc.co.uk/articles.php?cat_id=4
  • https://web.watfordjc.co.uk/articles.php?article_id=25
  • https://web.watfordjc.co.uk/articles.php?article_id=51
  • https://web.watfordjc.co.uk/articles.php?article_id=27
  • https://web.watfordjc.co.uk/articles.php?article_id=47
  • https://web.watfordjc.co.uk/downloads.php?cat_id=1
  • https://web.watfordjc.co.uk/news_cats.php?cat_id=6
  • https://web.watfordjc.co.uk/gpg_keys/239C495E.asc
  • https://web.watfordjc.co.uk/articles.php?article_id=71
  • https://web.watfordjc.co.uk/gpg_keys/9DE6BB74.asc
  • https://web.watfordjc.co.uk/articles.php?article_id=54
  • https://web.watfordjc.co.uk/articles.php?article_id=12
  • https://web.watfordjc.co.uk/articles.php?article_id=19
  • https://web.watfordjc.co.uk/articles.php?article_id=13
  • https://web.watfordjc.co.uk/gpg_keys/584AED57.asc
  • https://web.watfordjc.co.uk/articles.php?article_id=3
  • https://web.watfordjc.co.uk/articles.php?article_id=32
  • https://web.watfordjc.co.uk/photogallery.php?photo_id=3
  • https://web.watfordjc.co.uk/gpg_keys/8C4CEBB8.asc
  • https://web.watfordjc.co.uk/news_cats.php?cat_id=10
  • https://web.watfordjc.co.uk/news.php?readmore=7
  • https://web.watfordjc.co.uk/gpg_keys/71FA3812.asc
  • https://web.watfordjc.co.uk/gpg_keys/B6A00510.asc
  • https://web.watfordjc.co.uk/articles.php?article_id=17
  • https://web.watfordjc.co.uk/articles.php?cat_id=6
  • https://web.watfordjc.co.uk/articles.php?cat_id=2&rowstart=15
  • https://web.watfordjc.co.uk/articles.php?article_id=73
  • https://web.watfordjc.co.uk/articles.php?article_id=73&rowstart=1
  • https://web.watfordjc.co.uk/articles.php?article_id=73&rowstart=5
  • https://web.watfordjc.co.uk/articles.php?article_id=68&rowstart=2
  • https://web.watfordjc.co.uk/articles.php?cat_id=2&rowstart=45
  • https://web.watfordjc.co.uk/articles.php?cat_id=2&rowstart=30
  • https://web.watfordjc.co.uk/articles.php?article_id=69&rowstart=1
  • https://web.watfordjc.co.uk/articles.php?article_id=68&rowstart=0

Perhaps not surprisingly, there are zero Google results for www.watfordjc.co.uk because that domain is no longer resolvable and hasn't been for some time.

Google's Index of TheJC.me.uk

While I'm at it, I might as well include a full index of my other domains, since they contain much less content. Perhaps they might contain more content, as Google seems to have indexed some sub-domains that it shouldn't have.

  • https://jawc.aka.thejc.me.uk/news_cats.php
  • https://jawc.aka.thejc.me.uk/articles.php
  • https://jawc.aka.thejc.me.uk/search.php
  • https://jawc.aka.thejc.me.uk/weblinks.php
  • https://jawc.aka.thejc.me.uk/lostpassword.php
  • https://jawc.aka.thejc.me.uk/downloads.php
  • https://jawc.aka.thejc.me.uk/faq.php
  • https://jawc.aka.thejc.me.uk/photogallery.php
  • https://jawc.aka.thejc.me.uk/news.php?readmore=7
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=74
  • https://jawc.aka.thejc.me.uk/news.php?readmore=10
  • https://jawc.aka.thejc.me.uk/news.php?readmore=2
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=51
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=23
  • https://jawc.aka.thejc.me.uk/news.php?readmore=9
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=65
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=46
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=47
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=10
  • https://jawc.aka.thejc.me.uk/articles.php?cat_id=3
  • https://jawc.aka.thejc.me.uk/articles.php?article_id=12
  • http://mirrors.thejc.me.uk/smitfraudfix/SmitFraudFix.php
  • http://mirrors.thejc.me.uk/smitfraudfix/ScreenShot.php

Google's Index of JohnCook.co.uk

  • http://johncook.co.uk/
  • http://johncook.co.uk/about
  • http://johncook.co.uk/links
  • http://johncook.co.uk/articles
  • http://johncook.co.uk/music
  • http://johncook.co.uk/articles/security/gpg-keys
  • http://johncook.co.uk/blogs/website/redesign
  • http://johncook.co.uk/links/complete-sitemap
  • http://johncook.co.uk/blogs/politics/why-english-democrats-not-independent
  • http://johncook.co.uk/blogs/politics/not-standing-future-site-purpose
  • http://johncook.co.uk/archives/thejc-accessibility-2008
  • http://johncook.co.uk/archives/thejc-homepage-2006
  • http://johncook.co.uk/archives/watfordjc-homepage-2011
  • http://johncook.co.uk/archives/watfordjc-homepage-2014
  • http://johncook.co.uk/archives/watfordjc-homepage-2006
  • http://johncook.co.uk/archives/thejc-news-2008-2011
  • http://johncook.co.uk/archives/watfordjc-homepage-2006-R2
  • http://johncook.co.uk/articles/webhosting/problems-receiving-mail-using-cpanel

Starting with the Easy URLs

My GPG keys are located in the folder /gpg_keys, and Google currently considers https://web.watfordjc.co.uk their domain. The question is: where do I want the canonical URLs to be?

I believe https://Web.JohnCook.UK/downloads/gpg-keys/ would be the most logical place to store my keys. JohnCook.UK because it isn't unsuitable for work (i.e. suitable for work with the double negative removed), and in order to follow my site structure and URL naming conventions they belong in the downloads section and the 'gpg-keys' tag (if I were to create categories for downloads).

The advantage of starting here would be that the files already exist, although I will need to move them. Thus, I will need the following redirects in place:

  • https://web.watfordjc.co.uk/gpg_keys/239C495E.asc -> https://web.johncook.uk/downloads/gpg-keys/239C495E.asc
  • https://web.watfordjc.co.uk/gpg_keys/43F159F0.asc -> https://web.johncook.uk/downloads/gpg-keys/43F159F0.asc
  • https://web.watfordjc.co.uk/gpg_keys/4656E9D2.asc -> https://web.johncook.uk/downloads/gpg-keys/4656E9D2.asc
  • https://web.watfordjc.co.uk/gpg_keys/584AED57.asc -> https://web.johncook.uk/downloads/gpg-keys/584AED57.asc
  • https://web.watfordjc.co.uk/gpg_keys/6F84B8B4.asc -> https://web.johncook.uk/downloads/gpg-keys/6F84B8B4.asc
  • https://web.watfordjc.co.uk/gpg_keys/71FA3812.asc -> https://web.johncook.uk/downloads/gpg-keys/71FA3812.asc
  • https://web.watfordjc.co.uk/gpg_keys/8C4CEBB8.asc -> https://web.johncook.uk/downloads/gpg-keys/8C4CEBB8.asc
  • https://web.watfordjc.co.uk/gpg_keys/8FCF5FBB.asc -> https://web.johncook.uk/downloads/gpg-keys/8FCF5FBB.asc
  • https://web.watfordjc.co.uk/gpg_keys/B6A00510.asc -> https://web.johncook.uk/downloads/gpg-keys/B6A00510.asc
  • https://web.watfordjc.co.uk/gpg_keys/C5009D27.asc -> https://web.johncook.uk/downloads/gpg-keys/C5009D27.asc
  • https://web.watfordjc.co.uk/gpg_keys/C7F36802.asc -> https://web.johncook.uk/downloads/gpg-keys/C7F36802.asc

I will also need to modify my GPG Keys article and change the linked URLs to the new locations.

I now have two issues. The first is that the new domains are not yet functional. The second is that my GPG Keys article has been modified and therefore the OpenPGP sigature of the article is no longer valid.

To deal with the first issue, I need to create a new nginx configuration for web.johncook.uk.

Configuring NGINX For New Domain

In order to configure nginx I shall copy the existing configuration file and then make some modifications:

sudo su
cd /etc/nginx/conf.d/
cp johncook_co_uk.conf johncook_uk.conf
nano johncook_uk.conf

As I am using a naming convention for my SSL-related files, I simply need to change all instances of web.johncook.co.uk to web.johncook.uk, and all instances of 201404 to 201501.

The final thing to do is change my 307 temporary redirects in both johncook_co_uk.conf from https://web.johncook.co.uk$request_uri to https://web.johncook.uk$request_uri for SSL/SPDY hostnames johncook.co.uk and www.johncook.co.uk, thereby shunting https://johncook.co.uk and https://www.johncook.co.uk to https://web.johncook.uk.

For the time being I shall keep https://web.johncook.co.uk running as is, whilst gradually moving URLs to the new domain and introducing 301 permanent redirects.

As for redirecting the GPG key URLs listed above, that is a simple case of modifying my nginx configuration file and adding 301 redirects as follows:

rewrite ^/gpg_keys/239C495E.asc$ https://web.johncook.uk/downloads/gpg-keys/239C495E.asc permanent;

Well, I say it is as simple as that. I am going to need a line for every URL I am permanently redirecting, and my configuration files are not the easiest thing to navigate as I have previously mentioned.

Where Do I Need To Redirect From?

One big problem is that my sites have changed so many times over the years I don't even know which URLs are currently valid for these files. Time for another list:

  1. https://web.watfordjc.co.uk
  2. https://watfordjc.co.uk

Although watfordjc.com is no longer active, there are still Google results for the site because Googlebot has been assuming the https:// site has an invalid certificate, therefore keeping watfordjc.com alive in Google even though it is in essence just Googlebot accessing https://web.watfordjc.co.uk if it were to bother looking at the certificate (both domains use the same IP address).

One advantage of this is that when I create a redirect I am creating it not only for (web.)watfordjc.co.uk, but also watfordjc.com - killing two sites with one stone.

More Complex Nginx Redirects

Redirecting articles on watfordjc.co.uk from their old URLs to their new ones using nginx is rather more complicated.

In order to redirect I need to test the get arguments/parameters and, because I'm not redirecting them to similarly formatted URLs, I need to selectively redirect. Thus I have come up with the following:

location = /articles.php {
		if ($arg_article_id = "66") {
				rewrite ^.*$ https://web.johncook.uk/articles/security/gpg-keys? permanent;
		}
		if ($arg_article_id = "71") {
				rewrite ^.*$ https://web.johncook.uk/articles/security/spammers-and-spamming-domains? permanent;
		}
		proxy_pass	http://watfordjc.lighttpd;
		proxy_set_header	Accept-Encoding $spdy_ae;
		proxy_set_header	Host $host;
		proxy_set_header	X-Real-IP $remote_addr;
		proxy_set_header	X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_pass_request_headers	on;
}

There does not appear to be any way around copying and pasting the proxy stuff from the location / { } section, so it is either code duplication or having a 404 for all article_ids I haven't redirected yet.