Aside

Setting up internal domains with Pi-hole

This is a continuation of Self-hosting at home with Ubuntu Server + Easypanel on Mac Mini (late 2014).

The plan

What I did

Prep for installing Pi-hole on Ubuntu

In PowerShell, on my Ubuntu server, address the port 53 conflict with systemd-resolved:

  1. $user@192.168.x.x
  2. sudo sed -r -i.orig 's/#?DNSStubListener=yes/DNSStubListener=no/g' /etc/systemd/resolved.conf
  3. sudo sh -c 'rm /etc/resolv.conf && ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf'
  4. systemctl restart systemd-resolved
  5. Confirm that port 53 is free
    1. sudo lsof -i :53
    2. Nothing should return

Install, configure, and run Pi-hole with Easypanel

In Firefox, on my computer:

  1. Go to http://192.168.x.x:3000/
  2. Go to Dashboard > home project
    1. Select the ➕ to add an app
    2. Pick a service: App
    3. Service Name: pihole
    4. Select Docker Image
    5. Image: pihole/pihole:latest
    6. Select Save
  3. Expose some ports (for local network access)
    1. Go to pihole > Advanced
    2. Select Add Port
      1. Protocol: TCP
      2. Published: 8081
      3. Target: 80
      4. Select Create
    3. Repeat two more times for port 53
      1. Protocol: TCP & UDP
      2. Published: 53
      3. Target: 53
  4. Add Environment Variables
    1. Go to pihole > Environment
    2. Add Environment Variables:
TZ=America/Los_Angeles
FTLCONF_webserver_api_password=generatedPassword
FTLCONF_dns_listeningMode=ALL
FTLCONF_dns_upstreams=9.9.9.9;149.112.112.112
FTLCONF_misc_etc_dnsmasq_d=true

Note: FTLCONF_dns_listeningMode=ALL is essential for running Pi-hole in a Docker container. It can't listen to other devices on the local area network without it.

  1. Go to pihole and select Deploy, then Start
  2. In PowerShell, on my server, confirm Pi-hole is listening at port 53
    1. sudo lsof -i :53
    2. See docker-pr listed
  3. Back in Firefox, access Pi-hole's web GUI
    1. Go to http://192.168.x.x:8081/admin/login
    2. Log in with generatedPassword

Set router to use Pi-hole as DNS server

For my Asus router, in Firefox:

  1. Go to http://192.168.x.x
  2. Go to Advanced Settings > WAN
    1. Go to DNS Server and select Assign
    2. DNS Server 1: 192.168.x.x
    3. Select Save
    4. Select Apply
  3. Go to Advanced Settings > LAN > DHCP Server
    1. Go to DNS and WINS Server Setting
    2. DNS Server 1: 192.168.x.x
    3. Select Apply
  4. Disconnect from Mullvad VPN
  5. In PowerShell (new window), on my device, confirm Pi-hole is working
    1. nslookup ubuntu.com
    2. See Address: 192.168.x.x

(The Asus guide's screenshots don't match my router's settings, so I just did both. It works, so it's fine, right?)

Update Pi-hole settings & add internal domains

In Firefox:

  1. Go to http://192.168.x.x:8081/admin/
  2. Expand Settings on the left
  3. Go to Privacy
    1. Uncheck Query Logging
  4. Go to Local DNS Records
    1. Under List of local DNS records
      1. Add macmini.host.internal / 192.168.x.x
    2. Under List of local CNAME records
      1. Add jellyfin.home.internal / macmini.host.internal
      2. Add pihole.home.internal / macmini.host.internal
      3. Add miniflux.web.internal / macmini.host.internal
      4. Add easypanel.home.internal / macmini.host.internal

Update Easypanel app domains to internal domains

  1. Go to http://192.168.x.x:3000
  2. Select jellyfin
  3. Go to Domains > select Add domain
    1. Turn off HTTPS (since I don't have security certificates for this yet, but will look into this later)
    2. Host: jellyfin.home.internal
    3. Internal Port: 8096
    4. Select Create
    5. Make the new internal domain primary by starring it
  4. Go to http://jellyfin.home.internal/ (it works!)
  5. Update domain for pihole too
    1. Host: pihole.home.internal
    2. Internal Port: 80 (essentially, pick the port that Easypanel says the image exposes)
  6. Update domain for miniflux too
    1. Host: miniflux.web.internal
    2. Internal Port: 8080
  7. Go to Easypanel > Settings > General
    1. Custom Domain: easypanel.home.internal
    2. Select Save

Okay, now that everything works on the local network, I want everything to still work while I'm on Mullvad VPN.

Set Mullvad VPN to use Pi-hole as DNS server

  1. Connect on Mullvad VPN
  2. In PowerShell (new window), on my device, check DNS server IP
    1. nslookup ubuntu.com
    2. See that Address: is not 192.168.x.x
  3. Go to Mullvad VPN > Settings > VPN Settings
  4. Turn off all DNS content blockers
  5. Scroll down to Use custom DNS server
  6. Add a server: 192.168.x.x
  7. Disconnect from Mullvad VPN & reconnect
  8. In PowerShell, on my device, confirm Pi-hole is working
    1. nslookup ubuntu.com
    2. See Address: 192.168.x.x
  9. Go to http://jellyfin.home.internal/ (it works!)

Create security certificates for my internal domains

Because I don't like that Not Secure label next to my address bar & that one warning page that asks if I really want to proceed. A lot of recommendations to buy an example.com domain and use internal.example.com subdomain for internal stuff. But I don't want to. (Why do I have to rent a domain for this? I revolt.)

I want to set up security certificates for:

And I'll use mkcert:

  1. Install mkcert on my Ubuntu server by following the installation steps for Linux
    1. sudo apt update
    2. sudo apt install libnss3-tools -y
    3. curl -JLO "https://dl.filippo.io/mkcert/latest?for=linux/amd64"
    4. chmod +x mkcert-v*-linux-amd64
    5. sudo cp mkcert-v*-linux-amd64 /usr/local/bin/mkcert
    6. mkcert -install
    7. See:
Created a new local CA 💥
The local CA is now installed in the system trust store! ⚡️
  1. Create a directory for certificates
    1. sudo mkdir -p /etc/ssl/mkcert
    2. sudo chown $USER:$USER /etc/ssl/mkcert
    3. cd /etc/ssl/mkcert
  2. Generate certificates
    1. mkcert -key-file home-key.pem -cert-file home-cert.pem home.internal "*.home.internal"
    2. mkcert -key-file web-key.pem -cert-file web-cert.pem web.internal "*.web.internal"
    3. mkcert -key-file playground-key.pem -cert-file playground-cert.pem playground.internal "*.playground.internal"
  3. Confirm all the keys are there
    1. ls
    2. See: home-cert.pem home-key.pem playground-cert.pem playground-key.pem web-cert.pem web-key.pem

Configure security certificates in Easypanel + Traefik

Follow Easypanel's guide for Custom SSL Certificate Configuration:

  1. Move my certs & keys over to the directory that Easypanel wants them to be in
    1. cd
    2. sudo mkdir -p /etc/easypanel/traefik/certs
    3. sudo chown $USER:$USER /etc/easypanel/traefik/certs
    4. mv /etc/ssl/mkcert/*.pem /etc/easypanel/traefik/certs/
    5. ls /etc/easypanel/traefik/certs/
  2. Create the custom.yaml file:
tls:
  certificates:
    - certFile: /data/certs/home.crt
      keyFile: /data/certs/home.key
    - certFile: /data/certs/web.crt
      keyFile: /data/certs/web.key
    - certFile: /data/certs/playground.crt
      keyFile: /data/certs/playground.key
  1. Rename my files to match the desired naming convention:
mv /etc/easypanel/traefik/certs/home-cert.pem /etc/easypanel/traefik/certs/home.crt
mv /etc/easypanel/traefik/certs/home-key.pem /etc/easypanel/traefik/certs/home.key

mv /etc/easypanel/traefik/certs/web-cert.pem /etc/easypanel/traefik/certs/web.crt
mv /etc/easypanel/traefik/certs/web-key.pem /etc/easypanel/traefik/certs/web.key

mv /etc/easypanel/traefik/certs/playground-cert.pem /etc/easypanel/traefik/certs/playground.crt
mv /etc/easypanel/traefik/certs/playground-key.pem /etc/easypanel/traefik/certs/playground.key

ls /etc/easypanel/traefik/certs/
  1. Restart Traefik
    1. Go to http://192.168.x.x:3000/
    2. Go to Settings > General
    3. Select Traefik > Restart
  2. Update app domain settings to turn on HTTPS
    1. Go to jellyfin > Domains
    2. Select Edit jellyfin.home.internal
    3. Toggle on HTTPS
    4. Select Save
    5. Repeat for other apps
  3. Go to https://jellyfin.home.internal
    1. See Warning: Potential Security Risk Ahead
    2. Select Advanced...
    3. Select View Certificate
    4. See my mkcert certificate listed
    5. Close window

Install certificate on my computer

  1. Find the root certificate on my Ubuntu server
    1. ls ~/.local/share/mkcert/
    2. See it here: ~/.local/share/mkcert/rootCA.pem
    3. cat ~/.local/share/mkcert/rootCA.pem
    4. Copy the certificate's content (including BEGIN and END lines)
  2. Save it to my computer
    1. Open Notepad and paste content
    2. Save as mkcert-macmini.crt
  3. Install the certificate on my computer
    1. Double-click the .crt file
    2. Select Install Certificate
    3. Select Local Machine
    4. Select Place all certificates in the following store
    5. Select Trusted Root Certification Authorities
    6. Select Next
    7. Select Finish
    8. Select OK and OK again
  4. In Firefox, go to https://jellyfin.home.internal
    1. Still seeing: Warning: Potential Security Risk Ahead (checked in Edge browser & didn't get the warning)
    2. It turns out Firefox has a separate certificate store
  5. Add the certificate to Firefox
    1. In Firefox, go to about:preferences#privacy (this is a URL)
    2. Scroll down to Security
    3. Select View Certificates...
    4. Go to Authorities
    5. Select Import...
    6. Select mkcert-macmini.crt file
    7. Tick Trust this CA to identify websites.
    8. Select OK and OK again
    9. See my mkcert name listed
  6. Go to https://jellyfin.home.internal in Firefox (it works!)

Automate renewals for security certificates

Certificates expire after 90 days, and I cannot be relied upon to manually renew every three months, so I automate it with a cron job. This only needs to be set up with Traefik on the server & doesn't affect the devices we use.

  1. Create a new shell script
    1. sudo nano /usr/local/bin/renew-mkcert-certs.sh
  2. Write a script based on the steps I performed manually earlier
#!/bin/bash

# Paths
CERT_DIR="/etc/easypanel/traefik/certs"

echo "Generating new mkcert certificates..."

# Generate new certs
mkcert -key-file "$CERT_DIR/home-key.pem" -cert-file "$CERT_DIR/home-cert.pem" home.internal "*.home.internal"
mkcert -key-file "$CERT_DIR/web-key.pem" -cert-file "$CERT_DIR/web-cert.pem" web.internal "*.web.internal"
mkcert -key-file "$CERT_DIR/playground-key.pem" -cert-file "$CERT_DIR/playground-cert.pem" playground.internal "*.playground.internal"

echo "Renaming .pem to .crt/.key..."

# Rename files for Easypanel/Traefik
mv -f "$CERT_DIR/home-cert.pem" "$CERT_DIR/home.crt"
mv -f "$CERT_DIR/home-key.pem" "$CERT_DIR/home.key"

mv -f "$CERT_DIR/web-cert.pem" "$CERT_DIR/web.crt"
mv -f "$CERT_DIR/web-key.pem" "$CERT_DIR/web.key"

mv -f "$CERT_DIR/playground-cert.pem" "$CERT_DIR/playground.crt"
mv -f "$CERT_DIR/playground-key.pem" "$CERT_DIR/playground.key"

echo "Certificates renewed and placed in $CERT_DIR."
  1. Make the script executable
    1. sudo chmod +x /usr/local/bin/renew-mkcert-certs.sh
  2. Test if the script works
    1. /usr/local/bin/renew-mkcert-certs.sh
    2. See:
Generating new mkcert certificates...

[...]

Renaming .pem to .crt/.key...
Certificates renewed and placed in /etc/easypanel/traefik/certs.
  1. Confirm the new certificates are there
    1. ls -lh /etc/easypanel/traefik/certs
    2. See files & new timestamp:
-rw-r--r-- 1 $user $user 1.5K Apr 11 21:24 home.crt
-rw------- 1 $user $user 1.7K Apr 11 21:24 home.key
-rw-r--r-- 1 $user $user 1.5K Apr 11 21:24 playground.crt
-rw------- 1 $user $user 1.7K Apr 11 21:24 playground.key
-rw-r--r-- 1 $user $user 1.5K Apr 11 21:24 web.crt
-rw------- 1 $user $user 1.7K Apr 11 21:24 web.key
  1. Restart Traefik in Easypanel & confirm things still work as expected
    1. Go to http://192.168.x.x:3000/
    2. Go to Settings > General
    3. Select Traefik > Restart
    4. Go to https://jellyfin.home.internal
    5. No security warnings, we are good
  2. How do I restart Traefik via command line?
    1. See commands in Easypanel's guide for Traefik dashboard & docker service update--force can be used to perform a rolling restart without any changes to the service parameters
    2. docker service update --force traefik
    3. Test it out and see:
traefik
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service traefik converged
  1. Go to https://jellyfin.home.internal
    1. No security warnings, we are good
  2. Add Traefik restart into shell script
    1. sudo nano /usr/local/bin/renew-mkcert-certs.sh
    2. Add to the bottom: docker service update --force traefik
  3. Set script to run automatically
    1. crontab -e
    2. Since this is my first time using crontab, it wants to know my preferred editor — I select 1 for nano and press enter
    3. It opens a new file, and I add to the bottom: @monthly /usr/local/bin/renew-mkcert-certs.sh
    4. Save and see: crontab: installing new crontab
  4. One last time, confirm it all works
    1. /usr/local/bin/renew-mkcert-certs.sh
    2. ls -lh /etc/easypanel/traefik/certs
    3. See all the same confirmations as before & the new timestamps for the files
    4. It works!

Close exposed ports in Easypanel

Now that all the internal domains work, there's no need to access the apps at their IPs anymore. The exposed ports can be closed.

  1. Go to jellyfin > Advanced
    1. Select Remove for 8096
    2. Select Stop (⏹), Deploy, then Start (▶)
    3. Visit https://jellyfin.home.internal (everything still works)
  2. Repeat for miniflux
  3. I'm leaving pihole's ports open — 53 is essential for it to function, and for 8080, I want to still be able to access it via IP just in case (since this is the thing that makes the internal domains work)

References

I used ChatGPT to help me navigate the https + security certificates situation.

Notes