Warden Network Troubleshooting

Concrete failure signatures and validation order for warden0, VNET jails, host forwarding, default routes, and pf on FreeBSD.

Goal

Keep network debugging deterministic. The point is to identify whether the failure belongs to host bridge state, forwarding, jail routing, or pf, instead of guessing.

Canonical Model

Observed Failure Signatures

SymptomMeaning
ifconfig: interface warden0 does not existHost bridge is not present. Bastille cannot start the VNET jail.
jexec: jail "controlplane" not foundJail was never created, is not started, or Bastille start failed earlier.
route: route has not been found for defaultJail has no default route. Outbound traffic will fail even if the VNET interface is up.
fetch: ... Transient resolver failureUsually DNS is unreachable because routing or NAT is still broken.
pfctl: Table does not exist.Host pf config is incomplete for Warden. Bastille cleanup hit missing firewall state.
block drop all with no warden0 rulesHost firewall is blocking jail traffic before NAT or routing can succeed.

Validation Order

  1. Host bridge exists and has the gateway IP.
  2. Jail starts and has the expected VNET address.
  3. Host forwarding is enabled.
  4. Jail has a default route to 10.0.0.1.
  5. Jail can reach the host gateway.
  6. Host pf allows and NATs the Warden subnet.
  7. Only then test DNS and external APIs.

Canonical Checks

ifconfig warden0
sysctl net.inet.ip.forwarding
jls
sudo jexec controlplane /bin/sh -c 'hostname && ifconfig && netstat -rn'
sudo jexec controlplane /bin/sh -c 'ping -c 2 10.0.0.1'
ping -c 2 10.0.0.100
sudo pfctl -s info
sudo pfctl -sr

Persistent Host State

cloned_interfaces+="bridge0"
ifconfig_bridge0_name="warden0"
ifconfig_warden0="inet 10.0.0.1/24 up"
gateway_enable="YES"

These belong to host administration, not to jail-specific playbooks.

What We Learned

One-Shot Host Preparation

Use this when db or controlplane starts cleanly but cannot fetch packages or reach APIs.

sudo mkdir -p /usr/local/etc/bastille
sudo tee /usr/local/etc/bastille/resolv.conf >/dev/null <<'EOF'
nameserver 1.1.1.1
nameserver 9.9.9.9
EOF

# set in /usr/local/etc/bastille/bastille.conf
# bastille_resolv_conf="/usr/local/etc/bastille/resolv.conf"

# add to /etc/pf.conf
ext_if = "vtnet0"
warden_net = "10.0.0.0/24"

nat on $ext_if from $warden_net to any -> ($ext_if)
pass quick on warden0 inet from $warden_net to any keep state

sudo pfctl -nf /etc/pf.conf
sudo service pf reload
sudo bastille restart db
sudo jexec db /bin/sh -c 'fetch -qo- https://api.telegram.org >/dev/null && echo TELEGRAM_OK'