Computing/Running my own CA

I use the RFC-approved home.arpa domain at home. You cannot get certificates for devices from Let's Encrypt when you do this, because no-one (those of us who want certificates at least) has control over the DNS for home.arpa. This also means the CA has to be on the LAN, because a net-based service a) won't know how to look up home.arpa entries on the local router, b) and even if it could, it can't reach the internal services without firewall hole punching.

This page previously documented doing this in Docker. The general approach holds, but Docker is now replaced by Incus on my homelab server.

I ended up revisiting this work because I wanted to try and set up a local Forgejo instance with full registry support, enabling me to investigate Unifi's new OS Server blob without needing actual Docker.

Incus pieces

Add docker.io as an aliased remote for OCI.

Use incus create docker:smallstep/step-ca step-ca

config:
  image.architecture: x86_64
  image.description: docker.io/smallstep/step-ca (OCI)
  image.id: smallstep/step-ca
  image.type: oci
  limits.cpu: "1"
  limits.memory: 512MB
  oci.cwd: /home/step
  oci.entrypoint: /bin/bash /entrypoint.sh /bin/sh -c 'exec /usr/local/bin/step-ca
    --password-file $PWDPATH $CONFIGPATH'
  oci.gid: "1000"
  oci.uid: "1000"
devices:
  root:
    path: /
    pool: machines
    size: 10MB
    type: disk
  step-ca-config:
    path: /home/step
    pool: machines
    source: step-ca-config
    type: disk
ephemeral: false
profiles:
- livenetwork

CA Setup

1. Add the acme provider per the docs
   1. SSH to the Incus host
   2. incus shell step-ca
   3. step ca provisioner add acme --type ACME --x509-default-dur 730h # 1 monthish
2. Export out the root CRT file, will need it for trust everywhere
   1. certs/root_ca.crt
   2. Copy/paste works, or use the incus file commands to get the file out

An existing acme configuration can have the default duration extended by setting a parameter in ca.json.

 "claims": {                                                                      
   "defaultTLSCertDuration": "730h0m0s",                                    
   ...                                 
 },

Deploy

Caddy Reverse Proxy

1. Set up a basic Caddy2 service.
   1. incus create docker:caddy:alpine caddy --profile default --profile livenetwork --profile autoboot
2. Map three disk volumes to it

   devices:
     disk-device-1:
       path: /data
       pool: machines
       source: caddy-data
       type: disk
     disk-device-2:
       path: /config
       pool: machines
       source: caddy-config
       type: disk
     disk-device-3:
       path: /etc/caddy
       pool: machines
       source: caddy-etc
       type: disk
   

3. Add a Caddyfile with proxies for local services.
   1. As a migration method, disable auto-upgrade to TLS and reverse proxy both http and https to the local services (like Navidrome)
   2. Set up the local services with names like navidrome-c to indicate it's the container
   3. Use CNAME aliasing to alias the friendly names to the Caddy server so that it handles the requests and proxies them

   4. service.home.arpa {
              reverse_proxy service-c.home.arpa
      
              tls acme@home.arpa {
                      ca https://step-ca.home.arpa:9000/acme/acme/directory
                      ca_root /config/ssl/ca.pem
              }
      }
      
      # Explicit HTTP listeners without redirecting to HTTPS
      http://service.home.arpa {
              reverse_proxy service-c.home.arpa
      }

      Relies on DNS, but if that's broken the whole home network is broken.

/data holds all the important bits, like issued certificates. /etc/caddy holds the Caddyfile.

Most services are built from OCI/Docker containers and require DHCP. The -c workaround ensures the DHCP -> DNS registration done by Openwrt doesn't interfere with the TLS certificate issuing.

---

HomeAssistant

1. Update the http config to allow trusted reverse proxies
2. Install the Caddy2 add-on from https://github.com/einschmidt/hassio-addons
3. Add a Caddyfile that specifies the local CA
4. Ensure the Caddyfile points to the exported CA certificate in somewhere like /config
5. Start Caddy2 and it should successfully retrieve a certificate

Working Caddyfile

homeassistant.home.arpa {
	reverse_proxy homeassistant:8123
	tls acme@home.arpa {
		ca https://vault.home.arpa:9000/acme/acme/directory
		ca_root /config/ssl/ca.pem
	}
}

Working configuration.yaml

http:
  use_x_forwarded_for: true
  trusted_proxies:
    - 192.168.0.194  # whatever the homeassistant box resolves to

---

Desktop

1. For firefox, find the CA cert, add it to the local store

---

Router

1. Add router.home.arpa to the static DNS setup
2. Install the luci-app-acme UI; it pulls in acme.sh
3. Delete existing configs
4. Add new config pointing to IP address of vault as custom source, including the port
5. Hack the code to add --insecure to the curl call
6. Request name router.home.arpa
7. Should all work

/etc/config/acme

config acme
        option state_dir '/etc/acme'
        option debug '0'
        option account_email 'acme@home.arpa'
 
config cert 'router'
        option enabled '1'
        option use_staging '0'
        option keylength '2048'
        list domains 'router.home.arpa'
        option update_uhttpd '1'
        option validation_method 'webroot'
        option webroot '/www'
        option use_acme_server '1'
        option acme_server 'https://192.168.0.218:9000/acme/acme/directory'

The server option is fragile if the step-ca container is rebuilt totally and the old MAC address is not persisted.

Errors

• Curl error 35 - probably forgot the port number
• Curl error 60 - probably forgot --insecure

---

iOS

1. Email the root certificate to myself
2. Tap on it in Mail
3. Save to iCloud or phone
4. Open Files app
5. Tap certificate
6. Open Settings, should go to profile imports
7. Import the certificate
8. Settings root, search for trust
9. Enable trust for the certificate

Paths taken, but reversed

1. Use the letsencrypt add-on for HA, and specify the CA crt in the configuration. Have to apply https://github.com/home-assistant/addons/issues/2713's fix or things fail with error code 6.
2. Use the step-client add-on for HA. This got the root CA crt copied over, but didn't help with anything else really. Easier to just copy the certificate by hand.

Troubleshooting

At some point, my ACME provisioner vanished. This caused HomeAssistant's Caddy2 docker no end of grief, because it couldn't renew any more. The error message on the client side is very perplexing - urn:ietf:params:acme:error:accountDoesNotExist / "account does not exist". It's not saying that the email identifier of the client doesn't exist, but that the provisioner doesn't exist.

Root-caused - hadn't provided an outside-of-docker store for secrets etcetera, so an upgrade of the image lost all the generated configuration.

Other root-caused - acme.sh has cached credentials for the certificate, and now that account doesn't exist either. Happens when the filesystem resets post upgrade, and after restoring the acme provider.

It's easily fixed by re-adding the provisioner to the Step CA setup, but then a whole new set of CA certs may need distributing - I had to do this at least, and the dates on the certs had changed. Without doing this, Firefox will show SEC_ERROR_BAD_SIGNATURE.