Exposing services behind CGNAT with ZeroTier and Traefik
I self-host a few things for myself, friends, and family. I have an AdGuard Home server for DNS ad-blocking, an n8n node to automate some things like decrypting my credit card statements, a Jellyfin instance for the movie collection I never seem to have time for, among other things. Most of these run on a little HP ProDesk 600 G2 I bought used for $70 back in 2022, which has since been quietly running Debian 24/7, tucked away from sight in my living room.
These services worked perfectly while I was home, but accessing them while I'm away quickly became a real need, especially when I began self-hosting services like Actual which I had a real use-case for and didn't just host for fun. In an ideal world I could just point my domain to my home IP and be done with it, but my ISP puts everyone except business users behind a CGNAT, and exposing my IP didn't sound like an attractive idea regardless.
So I rented out a $5/mo VPS from OVH Singapore and tried a few different ways to make the services I host at home reachable over the Internet. Reverse SSH tunnels worked great but quickly became unmaintainable as I exposed more services. Cloudflare Tunnels also worked great, and are an easy option if you already use Cloudflare, but aren't ideal for tunnelling video services like Jellyfin. ngrok is another solution similar to Cloudflare, but custom domains are a paid feature, and having to log into https://1eb2-181-80-12-3.ngrok.app to access my budgeting app wasn't exactly an intuitive and memorable process.
ZeroTier as a tunnel
ZeroTier is a peer-to-peer VPN that puts all of your devices on the same virtual network, even if they're physically kilometers apart. I was already using ZeroTier to link my services and computers together, as it was way easier to remember that my servers had IPs 192.168.196.1 and 192.168.196.2 instead of 213.29.47.38 and 153.131.48.77. Crucially, although ZeroTier is a VPN, most of the time your traffic flows directly between your devices – ZeroTier only helps establish the direct link, barring any weird routing logic from your ISP. This means that traffic over ZeroTier is capped only by your (ISP) bandwidth.
I was only using ZeroTier to SSH into my servers more easily, and it didn't occur to me until recently that I could use it as a tunnel for exposing my services at home. Doing it via ZeroTier takes care of two things:
- IPs are sane. Even if my public home or server IP changes, ZeroTier addresses remain the same, and I don't need to manually re-establish the link, unlike with reverse SSH tunnels.
- Traffic remains private and uncapped. ZeroTier traffic is encrypted, and since it's not flowing through an intermediate cloud, I'm not restricted by bandwidth limits. By comparison, ngrok has a limit of 1GB/mo free outbound traffic, and although Cloudflare doesn't publish a hard cap, you can't push video or heavy media through a Cloudflare Tunnel.
What you'll need
Assuming you host services behind CGNAT on server A, you'll need (if you don't already have):
- A server
Bthat's accessible via public IP - A domain pointed to
B's IP address - A ZeroTier network that contains both servers
AandB
On both A and B, you'll need to get the physical interface name attached to your main IP. On A, this will be the interface that's connected to your home network, and on B, this will be the interface that's been assigned your public IP. You can get this using the ip command:
$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether ab:cd:ef:01:23:45 brd ff:ff:ff:ff:ff:ff
altname enp0s1f2The physical interface name here is eno1
On either A or B, you'll also need to get the virtual interface name of the ZeroTier network. To do this, run zerotier-cli listnetworks, and look for the string that starts with zt on your network row. For example:
$ sudo zerotier-cli listnetworks
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks abcdef1234567890 jared ab:cd:ef:01:23:45 OK PRIVATE ztabcdefgh 192.168.196.1/24The virtual interface name here is ztabcdefgh
Allow ZeroTier through your firewall
The first step is to allow the servers to talk to each other by allowing these rules in your firewall of choice:
- Inbound and outbound UDP traffic on port
9993(this is ZeroTier's default port) - Inbound and outbound UDP traffic on interface
ztabcdefgh
If you're using UFW like me, you can add these rules via these commands:
sudo ufw allow 9993/udp comment 'zerotier'
sudo ufw allow in on eno1 from any port 9993 proto udp comment 'zerotier - inbound'
sudo ufw allow in on ztabcdefgh comment 'zerotier - inbound iface'
sudo ufw route allow out on ztabcdefgh comment 'zerotier - outbound iface'Remember to change eno1 and ztabcdefgh to the interface names you got from the previous step. Do this on both servers A and B.
Add Traefik services
Setting up Traefik itself is beyond the scope of this document, so here I'm assuming you already have Traefik set up and accessible via *.example.com.
Suppose I want to expose Jellyfin over the domain jellyfin.dantis.me. I would need to add the following to Traefik's dynamic configuration on server B:
http:
routers:
jellyfin:
rule: Host(`jellyfin.dantis.me`)
service: jellyfin
middlewares:
- hsts
- streamingProxy
services:
jellyfin:
loadBalancer:
servers:
# Change this to your server A's ZeroTier IP
- url: http://192.168.196.1:8096
middlewares:
hsts:
headers:
customResponseHeaders:
Strict-Transport-Security: 'max-age=63072000; includeSubDomains'
# Magic fix for streaming performance via Traefik
streamingProxy:
headers:
customRequestHeaders:
Connection: ''Basically, configure Traefik like you normally would any other service, but specify your server A's full IP as the server endpoint for the load balancer.
Save your configuration, and if everything goes well, you should now be able to access your services on A via the host configuration you specified on B.
In summary
ZeroTier is a convenient tool for linking your services together on a single virtual mesh, and using it as a backhaul for my tunneled services is a no-brainer in hindsight. Now, whenever I need to expose a new service, I just need to add a couple of new blocks to Traefik's dynamic configuration, and it's already accessible in seconds.
Comments ()