Technical steps for recovering the Fairhaven Raspberry Pi remotely. For the overall approach see Troubleshooting and Home Assistant and the Raspberry Pi.

1. Purpose

This provides operational steps to:

Check system status remotely

Recover from common failure states

Roll back each hardening change if required

2. Remote availability checks (from Rowe or anywhere on tailnet)

There are three independent ways in. Tailscale is the normal route; ZeroTier One is a second, independent VPN kept as a fallback in case Tailscale fails; and the reverse SSH tunnel through the project VPS works over any internet connection, including the RUT240 mobile network, where both VPNs may be blocked by the carrier's address translation (CGNAT). A Nabu Casa subscription, if maintained, gives a fourth path to the HA dashboard only but is not currently maintained at the Fairhaven site. When each route is likely to work is set out in Troubleshooting.

2.1 SSH via Tailscale

ssh USER@<TAILSCALE IP ADDRESS>

2.2 Home Assistant

On-site LAN: http://<LOCAL IP ADDRESS>:8123

Via Tailscale/MagicDNS (if configured): http://<Tailscale MagicDNS>:8123

Via ZeroTier: http://<ZEROTIER IP ADDRESS>:8123

Via the reverse SSH tunnel: http://localhost:8123 (with the port forward in 2.4 running)

Via Nabu Casa (not currently maintained for the Fairhaven site): the Nabu Casa remote URL

2.3 SSH via ZeroTier One (if Tailscale fails)

ssh USER@<ZEROTIER IP ADDRESS>

2.4 SSH via the reverse SSH tunnel (if both VPNs fail)

The Pi keeps a permanent outbound SSH connection (autossh) to the project VPS, which opens a relay port there. Because the connection is outbound from the Pi it is not affected by CGNAT, so it works through the RUT240 mobile network once its mobile data has been turned on (SMS mobileon). Connect through the VPS, adding a local port forward if you also want the HA dashboard:

ssh -J VPSUSER@<VPS ADDRESS>:<VPS SSH PORT> -p <RELAY PORT> USER@localhost
ssh -J VPSUSER@<VPS ADDRESS>:<VPS SSH PORT> -p <RELAY PORT> -L 8123:localhost:8123 USER@localhost

3. On-Pi quick status snapshot (first action after login)

findmnt / -o SOURCE,FSTYPE,OPTIONS

systemctl is-active jf-remount-root-rw
systemctl is-active docker
systemctl is-active tailscaled
systemctl is-active zerotier-one
systemctl is-active autossh-tunnel

docker ps

HA local health: 401 is OK

curl -s -o /dev/null -w "HTTP %{http_code}\n" --max-time 5 http://127.0.0.1:8123/api/

journalctl -t jf_ha_healthcheck --no-pager | tail -n 50

4. Normal recovery actions (least to most disruptive)

4.1 Restart Home Assistant container only

docker restart homeassistant
docker logs --tail 150 homeassistant

4.2 Restart Docker (brings all containers back)

sudo systemctl restart docker
docker ps

4.3 Restart Tailscale (or ZeroTier, or the tunnel)

sudo systemctl restart tailscaled
tailscale status

sudo systemctl restart zerotier-one
sudo zerotier-cli info

sudo systemctl restart autossh-tunnel

4.4 Soft reboot (preferred reboot method)

sudo reboot

5. Failure playbooks

5.1 Pi reachable on LAN but not on Tailscale

Actions:

SSH on-site using LAN IP, then restart tailscaled and confirm DNS.

ssh USER@<LOCAL IP ADDRESS>
sudo systemctl restart tailscaled
tailscale status

getent hosts controlplane.tailscale.com
cat /etc/resolv.conf

If DNS looks wrong, confirm it is NetworkManager-generated (not Tailscale-generated).

5.2 Docker failed / containers not running

sudo systemctl restart docker
systemctl status docker --no-pager | sed -n '1,120p'
docker ps -a
docker logs --tail 200 homeassistant
docker logs --tail 200 mosquitto

Note:

Docker start-limits are disabled; it should not remain stuck failed.

5.3 Home Assistant is down but Linux is fine

First:

docker restart homeassistant

Then:

curl -s -o /dev/null -w "HTTP %{http_code}\n" --max-time 5 http://127.0.0.1:8123/api/

docker logs --tail 250 homeassistant

Also check the healthcheck actions:

journalctl -t jf_ha_healthcheck --no-pager | tail -n 80

5.4 Root filesystem came up RO again

Check:

findmnt / -o OPTIONS
dmesg | grep -i -E "ext4|journal|I/O error|read-only|uas|usb" | tail -n 160

Immediate recovery (if SSH is possible):

sudo mount -o remount,rw /
findmnt / -o OPTIONS

If RO events repeat, treat as a storage/power/USB transport issue to investigate separately.

6. Automated recovery behaviour (what happens without you)

Hardware watchdog reboots the host if Linux stops responding (systemd watchdog feed fails).

HA healthcheck runs every minute:

  • restart HA container after 3 consecutive failures (10 min cooldown)
  • reboot host after ~15 minutes of continuous failure (with restart grace period + reboot cooldown + loop protection)

7. Rollback procedures

7.1 Undo Tailscale DNS preference

sudo tailscale set --accept-dns=true
sudo systemctl restart tailscaled

7.2 Remove Docker override

sudo rm -f /etc/systemd/system/docker.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart docker

7.3 Remove container restart policies

docker update --restart no homeassistant mosquitto

7.4 Disable/remove the root remount helper

sudo systemctl disable --now jf-remount-root-rw.service
sudo rm -f /etc/systemd/system/jf-remount-root-rw.service
sudo systemctl daemon-reload

7.5 Disable/remove HA healthcheck (timer, service, script, state)

sudo systemctl disable --now jf-ha-healthcheck.timer

sudo rm -f /etc/systemd/system/jf-ha-healthcheck.timer
sudo rm -f /etc/systemd/system/jf-ha-healthcheck.service
sudo rm -f /usr/local/sbin/jf_ha_healthcheck.sh
sudo rm -rf /var/lib/jf_ha_healthcheck

sudo systemctl daemon-reload

7.6 Disable watchdog

Actions:

Remove dtparam=watchdog=on from the boot config file.

Remove RuntimeWatchdogSec / RebootWatchdogSec from /etc/systemd/system.conf.

Reboot.

8. Periodic maintenance checks (monthly)


Look for USB resets / IO errors that often precede RO boots

dmesg | grep -i -E "I/O error|reset SuperSpeed|usb reset|uas|ext4|journal|read-only" | tail -n 200

Confirm watchdog armed

systemctl show -p RuntimeWatchdogUSec -p RebootWatchdogUSec

Confirm HA healthcheck running

systemctl list-timers --all | grep jf-ha-healthcheck
journalctl -t jf_ha_healthcheck --no-pager | tail -n 60

<< Troubleshooting | | Water Infrastructure: tank, pump and feeder pipe >>      |Table of Contents>


Page last modified on November 08, 2026, at 09:29 AM