Moving a WordPress Site onto My Own Debian Server (Without Breaking Everything Else on It)

For years one of my sites lived with a shared hosting provider. Meanwhile I already had my own Debian server running a handful of other websites on Apache, with Let’s Encrypt certificates and a Postfix + Dovecot mail setup that I’d put together long enough ago that I no longer remembered how it worked.

Moving the site over sounded simple, and mostly it was. But there were three things I really wanted to get right:

  1. The website, a WordPress install, served over HTTPS, with plain HTTP redirecting to HTTPS.
  2. SSL for the new domain without touching any of the other sites’ certificates or configs.
  3. Exactly one email address for the new domain, able to both send and receive, without accidentally creating mailboxes for every other user on the server.

This post is everything I did, in the order I’d do it again, including the bits that tripped me up.

The names in this post

I’ve anonymised everything. Wherever you see these, substitute your own:

PlaceholderWhat it is
example.comThe server’s primary domain. Its mail host is mail.example.com
example.netAnother domain that was already hosted on the server
example.orgThe domain I was moving
dev@example.orgThe one and only mailbox the new domain should have
exampleorgThe Linux user that owns that mailbox
203.0.113.10The server’s public IP
/var/www/example.orgWhere WordPress is installed

Order matters

The one thing that shapes the whole process: Let’s Encrypt can only issue a certificate once the domain’s DNS points at your server. It checks that you own the domain by fetching a file over plain HTTP. So the order is:

  1. Set up the website, the mailbox and DKIM on the new server while DNS still points at the old host. Nothing breaks, and you can test most of it privately.
  2. Switch DNS.
  3. Get the certificate and turn on HTTPS.
  4. Test everything.

A day before switching, lower the TTL on your existing DNS records to something like 300 seconds. That way the switch takes effect in minutes rather than hours. And if your DNS is hosted by the old provider, move it to your registrar (or wherever you’ll keep it) before cancelling that account, or your domain will stop resolving entirely.


Step 0: Install WordPress the normal way

This post starts from a working WordPress install on the new server. Getting there is the standard WordPress-on-LAMP installation (Linux, Apache, MySQL/MariaDB, PHP), and I didn’t do anything unusual, so I won’t repeat it here. The official guide covers it well: How to install WordPress.

In short, on Debian:

  1. Install the stack, if the server doesn’t already have it. The command is below the list.
  2. Create a database and a database user just for this site, using sudo mysql.
  3. Download WordPress from wordpress.org and unpack it into the site’s folder, here /var/www/example.org. Then give Apache ownership with sudo chown -R www-data:www-data /var/www/example.org.
  4. Create a plain HTTP (port 80) vhost for the domain, then enable it with sudo a2ensite.
  5. Run the web installer, which creates wp-config.php and the admin account.

The packages for step 1:

sudo apt install apache2 mariadb-server php libapache2-mod-php php-mysql php-curl php-gd php-mbstring php-xml php-zip php-intl

One tip for step 5 when DNS still points at the old host: use the hosts-file trick from Step 1 and run the installer at http://example.org, not the server’s IP address. WordPress saves whatever address you install it from as its site address. Installing through the domain means Step 8 is a simple switch from http:// to https:// rather than a clean-up of IP addresses.


Step 1: Check the WordPress vhost

I’d already installed WordPress on the server with a plain port 80 virtual host. First I checked that Apache had picked it up correctly and nothing else was claiming the same name:

sudo apache2ctl -S | grep -i example.org

WordPress needs mod_rewrite for its pretty permalinks:

sudo a2enmod rewrite

Testing before DNS changes: add a line to the hosts file on your own computer so that the domain resolves to the new server for you alone:

203.0.113.10  example.org www.example.org

On Linux/macOS that’s /etc/hosts. On Windows it’s C:\Windows\System32\drivers\etc\hosts. Browse to the site, check it works, then remove the line so you don’t confuse yourself later.


Step 2: Work out how the mail server is actually set up

Mail setups come in several flavours: Linux system users, “virtual” mailboxes stored in files, or mailboxes in a database managed by something like PostfixAdmin. The steps for adding a domain are completely different for each, so first you have to find out which one you have.

These three commands answered it for me:

sudo postconf -n | grep -E 'mydestination|virtual_|smtpd_sender|milter|mydomain|myhostname'
sudo doveconf -n | grep -A4 -E 'passdb|userdb'
systemctl list-units --type=service | grep -Ei 'opendkim|rspamd|spamassassin|postfixadmin'

What they told me (trimmed):

mydestination = $myhostname, example.com, localhost.example.com, localhost, mail.example.com
virtual_alias_domains = example.net
virtual_alias_maps = hash:/etc/postfix/virtual
smtpd_milters = local:opendkim/opendkim.sock
passdb { driver = pam }
userdb { driver = passwd }
opendkim.service    loaded active running    OpenDKIM Milter

Decoded, that means:

  • Mailboxes are ordinary Linux users. Dovecot logs you in with your Linux username and password (PAM).
  • The primary domain is in mydestination, so every Linux user automatically receives mail at user@example.com.
  • Other domains are “alias domains”: Postfix accepts mail for them, then rewrites the address according to /etc/postfix/virtual.
  • OpenDKIM signs outgoing mail.

The existing /etc/postfix/virtual had exactly one line:

@example.net    @example.com

That’s a whole-domain rewrite. It turns anything@example.net into anything@example.com, which then lands in the Linux user called anything. That’s why every account on the server “worked” for the other domain.


Step 3: One mailbox, one address

I did not want to copy that wildcard for the new domain, for two reasons.

I only wanted dev@example.org to exist. With the wildcard, every Linux user on the server would get an @example.org address.

Wildcards cause backscatter. With @domain in the map, Postfix accepts mail for any name at that domain, including names that don’t exist, and only bounces it afterwards. Spammers fake the sender address, so those bounces land on innocent people, and your server’s reputation suffers. With an explicit address instead, mail to a name that doesn’t exist is refused while the sending server is still connected, and no bounce is ever sent.

The mailbox user

The email address and the Linux username don’t have to match. The virtual map connects them. I named the Linux user exampleorg rather than dev for two reasons:

  • It’s obvious what the account is for.
  • Every Linux user automatically gets an address on the primary domain, and here also on example.net through its wildcard. A user called dev would create dev@example.com and dev@example.net, and generic addresses like that are exactly what spammers guess.
sudo adduser --shell /usr/sbin/nologin exampleorg

The password you set here is the mail password. Because the account has no login shell, nobody can use it to SSH in.

Add the domain to Postfix

This appends the new domain to the existing list rather than replacing it:

sudo postconf -e "virtual_alias_domains = $(sudo postconf -h virtual_alias_domains) example.org"
sudo postconf virtual_alias_domains

Map exactly one address

Add one line to /etc/postfix/virtual:

dev@example.org    exampleorg

Then rebuild the lookup table and reload Postfix:

sudo postmap /etc/postfix/virtual
sudo systemctl reload postfix

Check the mapping. This should print exampleorg:

sudo postmap -q dev@example.org hash:/etc/postfix/virtual

Test it locally

swaks is the Swiss Army knife for SMTP testing (sudo apt install swaks). You can test delivery on the server itself before DNS has moved:

swaks --to dev@example.org --server 127.0.0.1

And an address that shouldn’t exist, which should be refused with 550 5.1.1 ... User unknown in virtual alias table:

swaks --to info@example.org --server 127.0.0.1

Gotcha: my first attempt used --server localhost and got Connection refused. On my server localhost resolves to the IPv6 address ::1 first, Postfix only listens on IPv4, and swaks gives up after the first address fails. Using 127.0.0.1 fixed it.

Check the mail log to confirm delivery. On newer Debian releases without rsyslog, use journalctl -u 'postfix*' instead:

sudo grep example.org /var/log/mail.log | tail -20

Can the mailbox send?

My main.cf had no sender restrictions. But the submission port (587) can have its own settings in master.cf, so it’s worth a look:

sudo grep -A15 -E '^(submission|smtps)' /etc/postfix/master.cf

No reject_sender_login_mismatch there, so the logged-in exampleorg user can send as dev@example.org straight away.


Step 4: DKIM with OpenDKIM

Without DKIM, Gmail and Outlook will happily send your mail to junk. First I checked how OpenDKIM was configured:

sudo grep -Ev '^\s*(#|$)' /etc/opendkim.conf

The important lines were:

KeyTable           refile:/etc/opendkim/key.table
SigningTable       refile:/etc/opendkim/signing.table

That’s the multi-domain layout. (If you only see a single Domain / KeyFile / Selector instead, OpenDKIM only signs one domain, and you’ll need to switch it to the table layout first.) The existing tables used the selector default and kept keys under /etc/opendkim/keys/<domain>/, so I followed the same pattern.

Generate a key:

sudo mkdir -p /etc/opendkim/keys/example.org
sudo opendkim-genkey -b 2048 -d example.org -D /etc/opendkim/keys/example.org -s default -v
sudo chown -R opendkim:opendkim /etc/opendkim/keys/example.org
sudo chmod 600 /etc/opendkim/keys/example.org/default.private

Add to /etc/opendkim/signing.table:

*@example.org      default._domainkey.example.org
*@*.example.org    default._domainkey.example.org

Add to /etc/opendkim/key.table:

default._domainkey.example.org    example.org:default:/etc/opendkim/keys/example.org/default.private

Leave trusted.hosts alone. It lists the machines allowed to send mail through OpenDKIM for signing, not domains. Then restart OpenDKIM:

sudo systemctl restart opendkim

My Postfix has milter_default_action = accept, so mail keeps flowing during the restart. Anything sent in that moment just goes out unsigned.

Finally, show the public key to put in DNS:

sudo cat /etc/opendkim/keys/example.org/default.txt

Step 5: DNS

Here’s the full set of records for the new domain. I copied the MX and SPF style from the domains already on the server.

TypeNameValue
A@203.0.113.10
Awww203.0.113.10
MX (priority 10)@mail.example.com
TXT@v=spf1 ip4:203.0.113.10 ~all
TXTdefault._domainkeyv=DKIM1; h=sha256; k=rsa; p=MIIB...
TXT_dmarcv=DMARC1; p=none; rua=mailto:dev@example.org

A few notes:

  • MX points at the server’s existing mail hostname. The mail server’s TLS certificate already covers that name, so I didn’t have to touch Dovecot or Postfix’s SSL config at all.
  • The DKIM file splits the key into several quoted chunks. Most DNS panels want it as one continuous string with the quote marks removed.
  • Remove any MX, SPF or DKIM records left over from the old host.
  • p=none in DMARC is deliberately gentle. It only asks receiving servers to send reports. Tighten it once everything passes.

Then wait until the domain resolves to the new server:

dig +short example.org

Step 6: A Let’s Encrypt certificate that doesn’t touch the other sites

The usual certbot --apache works, but it edits Apache configs for you. On a server with several other sites, I wanted certbot to do nothing except fetch a certificate. certonly --webroot does exactly that:

sudo certbot certonly --webroot -w /var/www/example.org -d example.org -d www.example.org

This creates a separate certificate under /etc/letsencrypt/live/example.org/. If certbot ever offers to expand an existing certificate belonging to another site, say no.

WordPress’s .htaccess rewrite rules leave real files alone, so certbot’s check file under .well-known/acme-challenge/ is served normally.

The trap to avoid: don’t write the port 443 vhost before the certificate exists. Apache refuses a config that points at certificate files that aren’t there, and if you reload with a broken config, every site on the server is affected, not just the new one. Certificate first, then vhost.

Also check the SSL module is on, and that certbot’s shared settings file exists. It will if other sites were previously set up with certbot’s Apache plugin:

sudo a2enmod ssl
ls /etc/letsencrypt/options-ssl-apache.conf

Step 7: The final vhost, with HTTP redirecting to HTTPS

Back up the current vhost first:

sudo cp /etc/apache2/sites-available/example.org.conf /root/example.org.conf.bak

Then replace its contents with this:

# Port 80: redirect everything to HTTPS, except Let's Encrypt renewal challenges
<VirtualHost *:80>
    ServerName example.org
    ServerAlias www.example.org
    DocumentRoot /var/www/example.org

    RedirectMatch 301 ^/(?!\.well-known/acme-challenge/)(.*)$ https://example.org/$1

    ErrorLog ${APACHE_LOG_DIR}/example.org_error.log
    CustomLog ${APACHE_LOG_DIR}/example.org_access.log combined
</VirtualHost>

# Port 443: WordPress
<VirtualHost *:443>
    ServerName example.org
    ServerAlias www.example.org
    DocumentRoot /var/www/example.org

    <Directory /var/www/example.org>
        Options -Indexes +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    SSLEngine on
    SSLCertificateFile    /etc/letsencrypt/live/example.org/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/example.org/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    ErrorLog ${APACHE_LOG_DIR}/example.org_error.log
    CustomLog ${APACHE_LOG_DIR}/example.org_access.log combined
</VirtualHost>

The redirect skips /.well-known/acme-challenge/, so automatic renewals keep working over plain HTTP without depending on the HTTPS side. If options-ssl-apache.conf didn’t exist, delete the Include line.

Always run configtest before reloading:

sudo apache2ctl configtest
sudo systemctl reload apache2

A reload is graceful. The other sites keep serving throughout.


Step 8: Tell WordPress it lives on HTTPS now

WordPress stores its own address in the database, and until you change it to https:// you’ll get redirect loops or broken logins. First I checked whether wp-config.php set the address:

sudo grep -E "WP_HOME|WP_SITEURL" /var/www/example.org/wp-config.php

It didn’t, so the address was in the database. To look at it you need the database name:

sudo grep DB_NAME /var/www/example.org/wp-config.php

(Yes, I first ran the next command with a literal DBNAME placeholder in it, and MySQL correctly told me that table doesn’t exist.)

sudo mysql -e "SELECT option_name, option_value FROM wordpress_exampleorg.wp_options WHERE option_name IN ('siteurl','home');"

Both came back as http://example.org, which is normal for a fresh install. There are two ways to switch it. Do this only after HTTPS works, otherwise you’ll send yourself to an address that doesn’t work yet.

Option A: set it in wp-config.php. Add these just above /* That's all, stop editing! */:

define('WP_HOME', 'https://example.org');
define('WP_SITEURL', 'https://example.org');

This is the most reliable way, because it doesn’t depend on being able to log in first. The downside is that the address fields in Settings โ†’ General become greyed out.

Option B: update the database directly. This keeps the fields editable:

sudo mysql -e "UPDATE wordpress_exampleorg.wp_options SET option_value='https://example.org' WHERE option_name IN ('siteurl','home');"

Pick one. If you’re migrating existing content rather than starting fresh, posts and images still contain the old http:// links. With WP-CLI you can fix them all at once:

sudo -u www-data wp search-replace 'http://example.org' 'https://example.org' --skip-columns=guid --path=/var/www/example.org

(The “Better Search Replace” plugin does the same from the dashboard.)


Step 9: WordPress’s own email

WordPress sends password resets and notifications from wordpress@example.org by default. Those messages go out through Postfix and are DKIM-signed, since the signing table covers *@example.org. But wordpress@ doesn’t exist as a mailbox, so any reply or bounce is refused.

First, set Settings โ†’ General โ†’ Administration Email Address to dev@example.org.

Then, to change the From address, I used WP Mail SMTP by WPForms. There are several plugins with similar names; this is the one whose author is listed as WPForms.

Skip the setup wizard

After activation it launches a wizard that insists you pick an “SMTP provider”: Gmail, SendGrid, “Other SMTP” and so on. You don’t need any of them. WordPress runs on the same server as Postfix, so there’s nothing external to connect to. Exit the wizard with the “Go back to the Dashboard” link at the bottom and go to WP Mail SMTP โ†’ Settings instead.

The settings

SettingValue
From Emaildev@example.org
Force From Emailโœ… ticked, so other plugins such as contact forms can’t override it
From Namethe site name
MailerOther SMTP
Return Pathโœ… ticked (“Set the return-path to match the From Email”)

Default (none) hands mail straight to the local Postfix, and OpenDKIM signs it for example.org automatically.

Return Path matters too. Without it, bounces go to the web server’s system account (something like www-data@example.com), where nobody will ever see them. With it ticked, they go to dev@example.org.

Other SMTP Setup

During the initial setup wizard (or from settings later), choose Other SMTP and point it at the local Postfix. That doesn’t need a login, because Postfix trusts connections from the server itself.

SettingValue
SMTP Host127.0.0.1
EncryptionNone
SMTP Port25
Auto TLSOff, because the mail certificate isn’t issued for 127.0.0.1, so TLS verification would fail
AuthenticationOff

Don’t be tempted to use mail.example.com:587 with the mailbox username and password instead. It works, but it stores the mailbox password in the WordPress database for no real benefit.

Test it

Go to WP Mail SMTP โ†’ Tools โ†’ Email Test and send a test to an outside address. In Gmail, “Show original” should show dkim=pass with d=example.org.

The plugin will keep nudging you towards its Pro version and a paid mailer service. You don’t need either.


Step 10: Test everything

Website. The first should return a 301 to https://example.org/, the second a 200:

curl -I http://example.org
curl -I https://example.org

Certificate renewal:

sudo certbot renew --dry-run

DKIM. You want key OK. key not secure next to it is normal; it just means your DNS doesn’t use DNSSEC:

sudo opendkim-testkey -d example.org -s default -vvv

Mail, for real:

  1. From an outside account such as Gmail, send to dev@example.org and check it arrives.
  2. From dev@example.org, send to the address shown on mail-tester.com. It reports SPF, DKIM and DMARC separately and tells you exactly what’s wrong if one fails. Gmail’s “Show original” gives a quick pass/fail too.

Mail client settings

Value
Incoming (IMAP)mail.example.com, port 993, SSL/TLS
Outgoing (SMTP)mail.example.com, port 587, STARTTLS, authentication on
Usernameexampleorg (the Linux user, not the email address)
Passwordthe one set with adduser

What I learned

  • Find out how your mail server works before changing anything. Five minutes with postconf -n and doveconf -n saved me from following a guide written for a completely different setup.
  • The email address and the mailbox don’t have to share a name. Postfix’s virtual map connects them, so you can give the Linux account a name that says what it’s for.
  • Avoid wildcard alias domains unless you really want every user on every domain. An explicit address gets unknown recipients refused immediately, and you avoid backscatter.
  • Get the certificate before writing the HTTPS vhost, and run apache2ctl configtest before every reload. A broken config affects every site on the server, not just the new one.
  • certbot certonly --webroot keeps certbot’s hands off your configs, and leaving /.well-known/acme-challenge/ out of the redirect keeps renewals working.
  • WordPress has to be told about HTTPS. Apache serving it over HTTPS isn’t enough.
  • If localhost gives “connection refused”, try 127.0.0.1. IPv6 strikes again.

Implement NFS Persistent Volume on Kubernetes

After setting up NFS in a previous post as well as using k3s to set up a lightweight Kubernetes cluster, we need to configure a Persistent Volume on Kubernetes to allow us to use the NFS storage in our containers.

The process to do this is as follows:

  1. Prepare a location for the Persistent Volume
  2. Create an NFS Persistent Volume
  3. Bind a Persistent Volume Claim to the Persistent Volume
  4. Create a Pod that mounts the Persistent Volume Claim
  5. Test the volume on the Pod

Prepare a location for the Persistent Volume

A Persistent Volume represents storage that is available for use in the cluster. There is no reason why we cannot create a single Persistent Volume that represents the entire NFS folder we have mounted, however, since several Pods will be storing data in that folder we might get conflicts if more than one of them creates a file with the same name. Consequently, it makes more sense to create Persistent Volumes for specific purposes.

To test our storage we will be using an Nginx container and the storage will keep the HTML files associated with the container. With that approach in mind we will be creating the following folders:

cd /mnt/nfs
mkdir pv
cd pv
mkdir nginx-html

Create an NFS Persistent Volume

The Persistent Volume (PV) we will be creating will be tied to our underlying NFS server. Since this volume will only be used to keep a few test pages we will be limiting the size of the volume to 50MB.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nginx-html
spec:
  capacity:
    storage: 50Mi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: pv-nginx-html  # storage class will be matched by the pvc
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /mnt/nfs/pv/nginx-html # path on the NFS server
    server: 192.168.0.151  # the NFS server

View this file on GitHub Gist

We configure the NFS server name as well as the path on the server where the volume will be mounted. The storage class name will later be used by the Persistent Volume Claim to bind to the Persistent Volume.

Deploy the Persistent Volume:

$ kubectl apply -f pv-nginx-html.yml
persistentvolume/pv-nginx-html created

Bind a Persistent Volume Claim to the Persistent Volume

A Persistent Volume Claim (PVC) is a request to use storage and in our case, we will be requesting the use of the Persistent Volume we have just created. It must also be noted that a PVC can also be dynamically provisioned using Storage Classes with a provisioner. This post is about Persistent Volumes, however, and consequently, we will be binding to the PV we have created.

This is the definition of our Persistent Volume Claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nginx-html
spec:
  storageClassName: pv-nginx-html  # k8s will look for a pv matching this storage class
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Mi # k8s will find a matching pv with at least this amount of storage

View this file on GitHub Gist

Kubernetes will match the Persistent Volume we created earlier using the storage class name and the storage request. It always attempts to find a Persistent Volume with at least the amount of storage requested.

Deploy the Persistent Volume Claim:

$ kubectl apply -f pvc-nginx-html.yml
persistentvolumeclaim/pvc-nginx-html created

Lets have a look at what has been created:

$ kubectl get pv,pvc
NAME                             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS    REASON   AGE
persistentvolume/pv-nginx-html   50Mi       RWX            Recycle          Bound    default/pvc-nginx-html   pv-nginx-html            23h

NAME                                   STATUS   VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/pvc-nginx-html   Bound    pv-nginx-html   50Mi       RWX            pv-nginx-html   23h

The Persistent Volume Claim has been bound to the Persistent Volume. It must be noted that a Persistent Volume can have only one Persistent Volume Claim bound to it.

Create a Pod that mounts the Persistent Volume Claim

We will be creating a Deployment with a volume. If you look at the YAML below you will notice that the spec includes a volume that specifies the PVC to use and the container then mounts that volume at the path specified.

Here is the deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-with-pvc-html
  labels:
    app: nginx-with-pvc-html
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-with-pvc-html
  template:
    metadata:
      labels:
        app: nginx-with-pvc-html
    spec:
      volumes:
      - name: nfs-html
        persistentVolumeClaim:
          claimName: pvc-nginx-html # name of the pvc being used as the volume
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nfs-html # name of the volume specified under 'volumes'
          mountPath: /var/www/html # path where the volume will be accessible in the container

View this file on GitHub Gist

Let’s deploy the Pod:

$ kubectl apply -f nginx-with-pvc-html.yml
deployment.apps/nginx-with-pvc-html created

Test the volume on the Pod

To test the volume we will connect to the container in the pod and then create a file. Afterwards, we will confirm that the file was created in our Persistent Volume path.

We will list the pods so that we can get our exact pod name:

$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
nginx-with-pvc-html-6fdb4bd6bd-lzlwv   1/1     Running   0          23h

Then we exec into the pod, go to the mounted path and create a test file. Afterwards, we exit again.

$ kubectl exec nginx-with-pvc-html-6fdb4bd6bd-lzlwv -it -- bash
$ cd /var/www/html
$ echo "<h1>This is a test index.html</h1>" >> index.html
$ exit

Lastly, we go the NFS path that was specified in our Persistent Volume and confirm that the file was created.

$ cd /mnt/nfs/pv/nginx-html
$ ls
index.html

Setting up a lightweight Kubernetes cluster with k3s, MetalLB and Nginx Ingress Controller

Inspired by this article of Scott Hanselman, we decided to build our own Kubernetes cluster but using the lightweight implementation of k3s and a few Raspberry Pis running Raspberry Pi OS. To wrap it all up we will install the Kubernetes Dashboard as well.

So what are we going to do?

  1. Prepare all the Raspberry Pis for installation of k3s
  2. Install k3s on all the Raspberry Pis
  3. Install the Kubernetes Dashboard

Prepare all the Raspberry Pis for installation k3s

To prepare the Raspberry Pis we need to do the following:

  1. Configure IP tables
  2. Disable swap
  3. Configure cmdline.txt
  4. Ensure all hostnames are unique
  5. Configure 64-bit kernel

These actions need to be done on each of the Raspberry Pis. Since these changes require a reboot to take effect it might make sense to perform your reboot once all four configuration steps are complete.

Configure IP tables

K3s currently does not support nftable-backed distributions and only (legacy) iptables we need to switch over to iptables (support for nftables is currently available in the Latest release channel of k3s, but not yet in the Stable release channel).

To switch to iptables we execute these commands:

$ sudo iptables -F
$ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
$ sudo reboot

Disable swap

Since swap creates performance problems with Kubernetes we need to disable swap before installation:

$ sudo dphys-swapfile swapoff 
$ sudo dphys-swapfile uninstall 
$ sudo update-rc.d dphys-swapfile remove

You need to reboot your Raspberry Pi for the change to take effect.

Configure cmdline.txt

We need to enable cgroups correctly for Kubernetes. Open /boot/cmdline.txt in an editor.

$ sudo nano /boot/cmdline.txt

We need to add the following to the end of the line in /boot/cmdline.txt. Make sure you don’t add a newline at the end.

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

You need to reboot your Raspberry Pi for the change to take effect.

Ensure all hostnames are unique

For the cluster to function correctly the hostnames of each of the Raspberry Pis need to be unique. The easiest way to do this is to use the raspi-config utility.

$ sudo raspi-config

Changing the hostname will require a reboot of the Raspberry Pi.

Configure 64-bit kernel

Since many services/containers out there are built targeting the 64-bit ARM processor we may want to set our kernel to run in 64-bit as well. This is optional, but if you decide to switch over to 64-bit at a later date you will have to re-install k3s. Ideally we want to use a full 64-bit OS, but at this time Raspberry Pi OS 64-bit is still only in beta.

To run the kernel in 64-bit we want to edit /boot/config.txt and add the following at the end of the file.

arm_64bit=1

After editing the file ensure that you reboot the device.

Install k3s on all the Raspberry Pis

We will be installing k3s from the Stable channel. For this example, we will be installing a server with two agent nodes (at the time of writing, the current Stable release of k3s only allows for a single server, but the Latest release has support for several server nodes).

Note: k3s uses the terms “server” & “agent” nodes, instead of the traditional “master” & “worker” nodes, however from the kubectl outputs you will still see reference to the “master” role.

Also, note that we will be using MetalLB as a load-balancer and Nginx as ingress controller.  We’ve found that in general those two are more widely used than the defaults installed by k3s.

Our process will be as follows:

  1. Install the k3s server node
  2. Install the k3s agent nodes
  3. Intall Helm
  4. Install MetalLB
  5. Install Nginx as ingress controller

Install the k3s server node

Firstly, we need to install k3s on the server node:

$ curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --no-deploy servicelb --no-deploy traefik
  • --write-kubeconfig-mode allows for writing to the kubeconfig file.
  • --no-deploy servicelb ensures the k3s doesn’t install a load-balancer, since we will be installing MetalLB later in the process.
  • --no-deploy traefik ensures that k3s doesn’t install Traefik as ingress controller. We will be using the Nginx ingress controller.

Install the k3s agent nodes

We need to obtain the server token so that we can use it to set up the agent nodes. The server token is obtained from /var/lib/rancher/k3s/server/node-token:

$ sudo cat /var/lib/rancher/k3s/server/node-token
K10f491d84da33f3080b547ad74395ffca7d9615bfa7213f9da13f922a034e5d1fb::server:d326b9ae03af3670a20b32a49aa9fc5d

Then we install each of the agent nodes. The format of the command is as follows:

curl -sfL https://get.k3s.io | K3S_URL=https://masterNodeIP:6443 K3S_TOKEN=servertoken sh -

By including the server token k3s will install an agent node and add it to the cluster. So for our example, we will execute the command as follows:

$ curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.151:6443 K3S_TOKEN=K108ef5976bbb1245c1fe4d2fc3df7b03b49a7bc5613436be7941c28260bdd571d4::server:ef80265b6ecc670f24f15ae8d2e817c3 sh -

After running the scripts confirm that all the nodes are added to the cluster by listing the nodes from the server node.  Keep in mind that initial startup and adding of agent nodes may take a second or two.

$ kubectl get nodes

NAME        STATUS   ROLES    AGE   VERSION
pi01        Ready    master   12m   v1.18.9+k3s1
pi02        Ready    <none>   93s   v1.18.9+k3s1
pi03        Ready    <none>   5s    v1.18.9+k3s1

The Role of “master” indicates the server node.

You can deploy containers to the cluster. Tip: you can install kubectl on your development machine to make it easier.

Install Helm

Helm is a package manager for Kubernetes and provides and simplified way of installing applications that generally have more complicated configuration requirements. It uses charts that define the configuration and there is an official repository of charts available at artifacthub.io.

To install helm use the instructions available on the Helm documents page. We found it simplest to make use of the Helm installation script.

After installation we want to add the official charts repository using the following:

$ helm repo add stable https://charts.helm.sh/stable
$ helm repo update

A few using Helm commands to note:

  • Installing applications:
    helm install <install-as-name> <chart-name> --namespace <namespace>
    If you have properties you want to change during the installation you can include --set <property-name>=<new-value>
  • Uninstall applications:
    helm uninstall <install-as-name> --namespace <namespace>
  • List applications:
    helm list --namespace <namespace>

Install MetalLB

MetalLB provides a load-balancer for bare metal Kubernetes installations. This means that every time a service is installed of type LoadBalancer on Kubernetes, MetalLB will automatically allocate a virtual IP address to it.

We will be using Helm to install MetalLB. In our example, we will put the installation in the metallb namespace, but feel free to put this in any namespace you prefer (many prefer to use kube-system since it is so closely tied to the core cluster functionality).

Adjust the IP address range to whatever works best for you.

$ helm install metallb stable/metallb --namespace metallb \
--set configInline.address-pools[0].name=default \
--set configInline.address-pools[0].protocol=layer2 \ 
--set configInline.address-pools[0].addresses[0]=192.168.0.201-192.168.0.250

Once it is done installing you can can view the pods created for MetalLB:

$ kubectl get pods -n metallb
NAME                                        READY   STATUS    RESTARTS   AGE
metallb-speaker-8hzrz                       1/1     Running   6          5m33s
metallb-controller-96765d758-crn45          1/1     Running   1          5m33s
metallb-speaker-bwllc                       1/1     Running   3          5m33s
metallb-speaker-sq4wf                       1/1     Running   4          5m33s

Install Nginx as ingress controller

We will be using Nginx as our Kubernetes Ingress Controller, to route external traffic to services in the cluster.

Once again we will make use of a Helm chart to do the installation for us. You will notice we set the property to disable the default backend, since we don’t require it. We will make use of the nginx-ingress namespace, but feel free to use any namespace name you prefer.

$ helm install nginx-ingress stable/nginx-ingress --namespace nginx-ingress --set defaultBackend.enabled=false

Once complete we can view the deployed controller and service.

$ kubectl get pods -n nginx-ingress
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-86db4564b5-2fl7c   1/1     Running   0          21h

$ kubectl get svc -n nginx-ingress
NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress-controller   LoadBalancer   10.43.96.206    192.168.0.201   80:30835/TCP,443:31913/TCP   6d8h

Feel free to navigate to the external IP allocate to the service to see the HTTP 404 page served by Nginx, proving that it has been deployed.

Install the Kubernetes Dashboard

To view nice values and metrics of our cluster we can install the Kubernetes Dashboard.

We follow these steps:

  1. Install the dashboard
  2. Configure an admin user
  3. View the dashboard

Install the dashboard

To install the dashboard we use the recommended configuration provided by the Kubernetes Dashboard GitHub site.

$ kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml

Configure an admin user

We then need to configure an admin user. We create two yml files: one to create the Service Account (user) and one to create the role binding for the Service Account. Keep in mind these are example users for full access to the cluster and it may pose a security risk.

admin-user.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

admin-user-role.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

Then we apply these files to create the user:

$ kubectl apply -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml

Now we need to get the token of the created user so that we can user it to log into the dashboard

$ kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token
token:      eyJhbGciOiJSUzI1NiIsImtpZC......tCuPSd66tZ27UquTJBpzF-6A0y_U0qCoCt63aQD6RXw

View the Dashboard

You can view the dashboard by running kubectl proxy and then navigating to http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/. However, you will need to tunnel through to that port from our development machine, since this it is hosted on your server Raspberry Pi.

Alternatively we can get the IP address of the dashboard and tunnel to it directly:

$ kubectl get svc --namespace kubernetes-dashboard

We can then create an SSH tunnel from our host machine to the dashboard:

$ sudo ssh -L 9797:10.43.99.66:443 pi@192.168.0.151

Once the tunnel is established you can navigate to the tunnelled port (9797 in this example) from your browser. On the login screen enter the token and then you can view details on the dashboard (choose the namespace in the top toolbar):

Creating an NFS with your Raspberry Pi

Inspired by this article we decided to create our own Network File System using a Raspberry Pi.

The process is as follows:

  1. Configure the NFS share on your Raspberry Pi
  2. Connect to the share from client devices

Configure NFS on your Raspberry Pi

For the purpose of this post, we will be using a USB drive connected to the Raspberry Pi. We will also be using Raspberry Pi OS (formerly Raspbian) as the operating system on the device.

Configuration has these steps:

  1. Install the required dependencies
  2. Prepare a mount point for your NFS share
  3. Mount the drive to the mount point
  4. Expose the mount point using NFS

Install the required dependencies

To host an NFS share on your Raspberry Pi we need to install the nfs-kernel-server package. This package enables the NFS protocol.

sudo apt-get update
sudo apt-get install nfs-kernel-server

At this point we now want to enable and start our NFS service:

sudo systemctl enable nfs-kernel-server.service
sudo systemctl start nfs-kernel-server.service

Prepare the mount point for your NFS share

We need to create a path where the share will be mounted. For the purposes of this post, we will be using the path /mnt/nfs.

sudo mkdir -p /mnt/nfs

We need to ensure that we have proper permissions on this share, otherwise clients of the share won’t be able to write to it.

The below set of commands do three things: makes the pi user and group the owner of the mount point, ensures we have the required permissions on all the directories on the mount point, and lastly ensures that any files have the required permissions.

sudo chown -R pi:pi /mnt/nfs
sudo find /mnt/nfs/ -type d -exec chmod 755 {} \;
sudo find /mnt/nfs/ -type f -exec chmod 644 {} \;

Mount the drive to the mount point

We need to mount the drive to the prepared mount point, but before we can do that we need to identify our drive. First, we find the path to the drive:

sudo fdisk -l

This command will list all connected devices. Here is an example of the portion related to a USB drive:

Disk /dev/sda: 114.6 GiB, 123060879360 bytes, 240353280 sectors
Disk model:  SanDisk 3.2Gen1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start       End   Sectors   Size Id Type
/dev/sda1          32 240353279 240353248 114.6G  c W95 FAT32 (LBA)

We can mount the USB drive to the prepared mount point as follows:

sudo mount /dev/sda1 /mnt/nfs

We want to ensure that the mount is automatically configured when the device reboots. To do that we need to obtain a few more pieces of information. We need to identify the UUID related to this drive so that we can configure the mount. Take note of the UUID and TYPE using the command below so that we can use it in the mount configuration.

sudo blkid /dev/sda1

/dev/sda1: UUID="721B-EFD1" TYPE="vfat"

The last pieces of information we need are the identifiers related to the pi user. Take note of the UID and GID values.

id pi

uid=1000(pi) gid=1000(pi)....

To ensure the mount is configured upon startup we add an entry to the /etc/fstab file. Start editing the file:

sudo nano /etc/fstab

Then add the required line to configure the mount. Use the UUID, UID and GID values we obtained previously. The UUID identifies the device to mount and the UID & GID defines what user & group to use for the mounting (this is especially important if your drive uses Windows-based file systems like FAT). The format of the entry is as follows:

UUID="<UUID of Device>" <path to mount point> <device filesystem type> <options> <freq> <passno>

The filesystem type we got from our blkid command previously and the options include the UID & GID we noted. We also configure some other options that affect the behaviour of the share. Consequently the entry we create for our example is:

UUID="721B-EFD1"    /mnt/nfs    vfat    rw,nosuid,nodev,nofail,noatime,uid=1000,gid=1000 0 0

Expose the mount point using NFS

The NFS server reads the /etc/exports file to know which paths to share. We need to add the path of our drive’s mount point using this file.

sudo nano /etc/exports

The basic format of the entry we need to add is as follows:

<path to share> <IP address that may access>(<options>)

The IP address value may be a mask that allows a range of IP addresses access. The options include the UID & GID we identified earlier so that our pi user is used whenever a client accesses the NFS share. We also configure some other options that help with performance and sets up user security correctly. Consequently, our entry looks like this:

/mnt/nfs    192.168.0.0/16(rw,all_squash,insecure,async,no_subtree_check,anonuid=1000,anongid=1000)

At this point we now want to restart our NFS service in order for the changes to take effect:

sudo systemctl restart nfs-kernel-server.service

Connect to the share from client devices

We can connect to the share from various other client devices. For this post, we will look at these clients:

Connect Windows machines to the NFS share

To connect a Windows machine to the NFS share we first need to enable the NFS client. Go to the Windows start menu, type “Windows Features” and then select “Turn Windows features on or off” from the menu.

Then enable all the options for “Services for NFS”:

Once that is done you can map the NFS share as a network drive in Windows Explorer. Click on “This PC” and then go to the “Computer” tab.

Lastly capture the path to the NFS share with the Raspberry Pi’s IP address:

Connect from Debian-based systems

To connect to the NFS share from Debian-based systems, like Ubuntu or Raspberry Pi OS, we first need to install the libnfs-utils package.

sudo apt-get install libnfs-utils

We need to create a mount point for the share:

sudo mkdir /mnt/nfs

Then we simply mount to the share:

sudo mount 192.168.0.151:/mnt/nfs /mnt/nfs

To ensure that the mount is configured again on reboot we add it to the /etc/fstab file (like we did on the NFS host).

192.168.0.151:/mnt/nfs /mnt/nfs nfs auto 0 0

Preparing a Headless Raspberry Pi

Oftentimes you only want to use a Raspberry Pi without any monitor or keyboard, known as running the Raspberry Pi headless. This post will focus on achieving that with Raspberry Pi OS (formerly known as Raspbian).

The basic steps are:

  1. Install Raspberry Pi OS
  2. Enable SSH
  3. Configure your wireless network (optional)
  4. Connect to your Raspberry Pi
  5. Update your Raspberry Pi

Install Raspberry Pi OS

You will need to install your OS of choice on the MicroSD Card used by your Raspberry Pi. If your computer does not have a slot for SD cards you will need to acquire an adapter first, so that you can access the SD card via another way, such as the USB port of your computer.

The easiest way to install Raspberry Pi OS is to use the official Raspberry Pi Imager. Since your Raspberry Pi will be running headless it probably makes sense to use the light-weight version of the OS called “Raspberry Pi OS Lite (32-bit)”, so make sure to choose that option. After specifying the OS and selecting your SD card you can click on the “Write” button. The imager will write to your SD Card and afterwards verify that everything was written successfully.

Note: Keep in mind that anything that is present on the MicroSD card will be erased during the write process of the imager.

Enable SSH

Since you won’t be using a monitor or keyboard you need to access your Raspberry Pi remotely. The best way to do this is by using SSH. In order to enable SSH on your Raspberry Pi, you need to create a file called ssh in the boot folder of your MicroSD card. The file is simply an empty file without any extension in the filename. When your Raspberry Pi boots up it will enable SSH and remove the file.

You create the file from your Windows or Mac machine, but if you have just finished writing the OS using the Raspberry Pi Imager you may need to remove your card and put it back in since the Imager may unmount the card after writing to it.

Configure your wireless network

If you need to have access to a WiFi network you will have to configure the wireless access as well. Just like enabling SSH, this is done by creating a file called wpa_supplicant.conf in the boot folder of your newly written MicroSD card before it starts up the first time. When your Raspberry Pi boots up it will copy the file to the correct path to configure the network.

Here is an example of the wpa_supplicant.conf file:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=<Specify your ISO 3166-1 country code>

network={
 ssid="<The SSID (Name) of your WiFi network>"
 psk="<The password of your WiFi network>"
}

Connect to your Raspberry Pi

After configuring everything you can now boot up your Raspberry Pi. If you don’t make use of a WiFi network you will need to connect to the network using an ethernet cable.

Once it has booted up you need to identify the IP address of your Raspberry Pi. You can do this by having a look on your router or making use of a mobile app like Fing to scan your network.

Once you have the IP address you can use whichever SSH client you prefer to connect to your Raspberry Pi. By default the username is pi and the password is raspberry.

ssh pi@192.168.0.151
pi@192.168.0.151's password:

The first time you connect to the Raspberry Pi you will also be prompted to confirm the fingerprint of the SSH certificate. Confirm by entering “y” for yes.

Update your Raspberry Pi

Now that your newly setup Raspberry Pi is up and running it would be a good idea to do a full update.

First we update the apt tool to have the latest package list:

sudo apt update

Then we trigger the full upgrade:

sudo apt full-upgrade

Triggering full-upgrade will ensure that all packages and their dependencies are updated. This same command will also update the kernel and firmware of your Raspberry Pi.