Specter PoE unit test — S1 + S2

Updated 2026-08-14 after storage catch-all. Cisco 192.168.86.21 · VMS 192.168.86.54:8788. Cameras left UP. MQTT storage_health=OK is not overlay truth. No secrets.

Why S2 storage was silent — then the catch-all

You did not miss a toast. Nothing in the existing watchers treats camera overlay as a first-class alert.

Watcher	Cadence	What it actually watches	Why 97% overlay was invisible
Platform Watcher (launchd, running)	12s tick · edge probes 30s · disk 5m	Host `/` on this Mac, MQTT keys present, go2rtc from 87.x	`disk_host` is the Mac disk (failCount 0). `edge_s1/s2` circuits are open because this Mac cannot L2-probe streams — 11k fails, not storage.
Soft Watcher (canvas)	12s	Stream repair via proxy MCP	No overlay probe. Escalates blue-LED / firmware-replay only.
MQTT `devices/*/health`	~health period	`recorder.storage_health`	Still published OK on S1 and S2 at 14:42 CDT after S2 had been 97%. Field is untrusted.
Edge recorder prune	retention / --prune-first	oldest `chunk_*.h265`	Does not delete `pre_event_thermal_*` (the 42 GB bomb) or 13 GB `logcat.log`.
Reconcile LaunchAgent	90s (if loaded on 86.54)	ghost_core missing → runc start	No-op when pipeline is up and disk is dying.

So MQTT was green, the host Watcher was yellow for the wrong reason, and no playbook pruned thermal pre-event files. That is the gap.

New catch-all scripts/fleet-catch-all.py (LaunchAgent com.ghostprotocol.specter-fleet-catchall):

Every 15s: read VMS /fleet/health MQTT payloads (storage_health, thermal, chunk path, age).
Every 30s: SSH df /overlay, count pre_event_thermal_*, measure logcat.log. This is the truth.
Decide (no LLM): overlay ≥80% warn, ≥92% or <1.5 GB free critical; pre_event ≥200; logcat ≥512 MB. storage_health=OK never overrides df.
Mitigate: prune pre_event keep 30, truncate logcat, drop caches. Cooldown 10 min/device. Ghost stays up.
Notify: macOS notification + ~/.specter-vision/platform/fleet-catchall.log + last-action JSON. Playbook id repair-edge-storage.

Not a 30s Grok/Hermes loop — Watcher doctrine is deterministic probe → playbook → verify. An LLM can explain after the fact; it must not be the 15s decision path.

Live health — 2026-08-14 14:12 CDT

Read-only jump + VMS. Both last-known oakapps still running. Grades are for this snapshot, not the boot clocks.

Nominal ops

	S1	S2
Uptime / load	8h09 · 2.26 2.30 2.21	8h09 · 2.28 2.14 2.20
CID / runc	`aaf26107…` running · 1624	`65acd4e6…` running · 1719
PIDs	agent 1771 · core 1812 · go2rtc 1775	1794 · 1903 · 1802
Layer	`layer1_rfdetr_thermal_safe` · `ENCODE_ONLY=0` · H.265 1280×720 · `H265_FPS=5` · `RF_DETR_STRIDE_FPS=5` · 1500 kbps · keyframe 2s
Canary	token match	token match
MQTT 86.54	shadow1 ~19s	shadow2 ~8s
VMS	fleet_grade 90 · readiness B/84 · Flask warn (host, not camera)

Ops: A− — last-known up. Not A: SoC warm/critical + H.265 FIFO stalls.

RF-DETR

	S1	S2
DLC	`ghost_protocol_transit.dlc` 135 MB + rfdetr_small	same
`.latest_dets.json`	age 0.2s · n=0 · persons=0 · 69.6°C normal	age 0.2s · n=0 · persons=0 · 78.8°C warm
`.rfdetr_status`	key=value (not JSON) · last_detection now · 0/min	same
ghost_core SoC	78–90.7°C CRITICAL spikes	79–82°C warm · VIDEO_FIFO drops ~355k
sysfs / MQTT thermal	63°C / mqtt 73.1°C — different sensors	56°C sysfs
Persons seen	MQTT health person_count=3 same minute (empty scene at sidecar snap is ok)	0 this snap

RF-DETR: B+ — file + live writer + MQTT persons. Not A: S1 SoC critical, warm stride×2, status file not JSON.

Storage

	S1	S2
`/overlay` at snap	62% · 48/77 GB	97% · 74.7/77 GB · 2.3 GB free
`/overlay` after prune	—	26% · 20.1/77 GB · 56.9 GB free
Today `2026/08/14`	2075 `.h265` · 5 `.tmp`	2534 `.h265` · 1 `.tmp`
`pre_event_thermal_*`	1321 files · 14.6 GB	2536 files · 42.3 GB (~20 MB / 10s)
justified `chunk_*`	756 · 11.4 GB	present (~60s); drowned by pre_event
root `/`	88% · 608 MB free	61%

Storage: S1 C+ · S2 snap F → after prune B. Deleted 2607 pre_event_thermal_* (43.5 GB, kept 30) while oakapp stayed up. df stayed 100% until logcat.log (13.3 GB) was truncated in place and caches dropped — then 26% / 56.9 GB free. Ghost still 1794/1903. Do not PoE-cut; open .tmp remains.

Recordings / FPS

Target from env is 5 fps. ffprobe reports 25/1 with no duration — ignore. Trust chunk sidecar + NAL / frame_seq.

Measure	S1	S2
Closed/open proof	`chunk_1786734726_980` sidecar	open `chunk_1786734751_199` dets.jsonl
Wall duration	60.38 s (target 60)	~55.5 s dets window
Pictures / ffprobe frames	412 = `nb_read_frames` 412	—
Sidecar `fps` tag	15.0 (stale vs env 5)	—
Measured wall fps	6.82	~4.86 (seq 142522→142792)
Session encoded_frames / 8.15 h	~6.9 fps	~4.85 fps
RF-DETR samples	38 dets lines / ~14 s on next chunk	152 lines on open tmp
Justified / VPS+IDR	yes	writing

Encode: B — continuous justified H.265. Actual ~5–7 fps, not 15 and not 25. Thermal stalls keep it off the sidecar 15 tag.

Run4 — 6:00am CDT one-shot simultaneous power-on

Scheduled one-shot. Login OK · adminEnable=1 both · T0 1786705338. 0 W right after enable (SOC boot). No reconcile. Good CIDs only: S1 aaf26107…4453 · S2 65acd4e6…9628c. New PIDs S1 ghost 1771/1812 (runc 1624) · S2 1794/1903 (runc 1719). Layer layer1_rfdetr_thermal_safe · ENCODE_ONLY=0. Scheduler 019ffea791e1 deleted after enable so it cannot fire again.

Gate	S1	S2
T_ping / T_ssh	00:52 / 00:31	00:52 / 00:52
T_layer / T_config	00:31 · token match	00:52 · token match
T_ghost / T_rfdetr	00:52	00:52
T_rec_start (first H.265)	01:59	01:59
T_rec_stable	02:42	02:41
T_nominal	02:42 · 3 min	02:41 · 3 min
Overlay now	41%	40%
Open `.h265.tmp`	5	1

CBS watts after detect=3 (post-nominal samples; milliwatts ÷ 1000). S1 sawtooth continues. S2 ~8–10 W.

T+	gi1 S1	gi2 S2	Aligns with
00:00	0 W · en=1 det=2	0 W · en=1 det=2	enable; SOC boot
03:38	4.6 W	7.9 W	post T_nominal
04:03	5.7 W	9.1 W
04:27	7.7 W	10.5 W	S2 peak this set
04:51	6.7 W	10.3 W	watch sample end

T_mqtt at 00:31 is stale — VMS still listed shadow1+shadow2 grade 90 across the overnight halt. Do not treat the device-name list as a fresh clock. Real pipeline proof is ghost_core + new H.265 (chunk_1786705383 / chunk_1786705389).

Cameras left UP. No halt. No PoE-cut. JSONL: docs/grading/poe-unit-test-s1-s2-run4.jsonl.

Run3 — simultaneous power-up (prior practice)

Both ports were 0 W / adminEnable=2. Enabled together (adminEnable=1, status 0). No reconcile. Good CIDs only. New PIDs S1 1763/1846 · S2 1699/1862.

Gate	S1	S2
T_ping / T_ssh	00:37	00:38
T_layer / T_config	00:37 · token match	00:38 · token match
T_ghost / T_rfdetr	00:60	01:00
T_rec_start (first H.265)	02:04	02:03
T_rec_stable	02:47	02:46
T_nominal	02:47 · 3 min	02:46 · 3 min
Overlay now	41%	72%
Open `.h265.tmp`	4	5

CBS watts after detect=3 (full 5 min watch). S1 is a sawtooth, not the old flat ~4.6 W note. S2 sits ~8–13 W.

T+	gi1 S1	gi2 S2	Aligns with
01:06	7.0 W	8.9 W	ghost_core just up
01:26	6.8 W	11.7 W	RF-DETR warm
01:45	7.8 W	11.9 W	pre first chunk
02:04	6.1 W	7.9 W	T_rec_start
02:24	7.1 W	12.3 W
02:43	7.5 W	8.6 W	T_nominal
03:02	3.6 W	8.6 W	S1 dip
03:22	5.9 W	12.8 W	S2 peak
03:41	2.9 W	8.3 W	S1 low
04:00	7.2 W	10.6 W
04:20	6.2 W	11.2 W
04:39	3.4 W	10.4 W
04:58	3.1 W	9.3 W	watch end

T_mqtt at 00:37 is stale — VMS still listed shadow1+shadow2 grade 90 from before the halt. Do not treat the device-name list as a fresh clock. Real pipeline proof is ghost_core + new H.265.

When it is safe to begin shutdown

Now is a good time to start the graceful script — both units are past T_rec_stable. It is not safe to PoE-cut this second: 4/5 temp chunks are still open.

Window	Do this	Do not
T=00:00 → T_ghost (~60 s)	Wait. OS / oak-agent coming up.	Halt or PoE-cut — messy files, possible blue-LED next boot.
T_ghost → T_rec_start (~60–124 s)	Wait unless emergency. Pipeline opening first file.	PoE-cut while encoder starts.
After T_rec_stable (now)	Run `scripts/edge-safe-shutdown.sh S1` then S2 (or both). Script: runc kill → wait ghost gone (~4–5 s) → `shutdown -h` → wait SSH dead (~6–9 s) → then Cisco `adminEnable=2`.	Skip the script and pull PoE. Halt leaves ~4–6 W until PoE off.
Emergency / no SSH	Cisco `adminEnable=2` only. Expect orphan `.tmp`.	Call that a clean stop.

Rule: inform the camera first, cut power last. The camera must close H.265. Cisco 0 W is step 4, not step 1. VMS 86.54 stays up.

Run3 shutdown executed 2026-08-14 after T_nominal. Parallel halt, then PoE.

Step	S1	S2
Pre PIDs	1763 / 1846	1699 / 1862
Pre overlay	41%	72% (1968 chunks)
ghost gone	+4 s	+8 s
Prune	—	1938 deleted, keep 30; df still 72% (space elsewhere or delayed reclaim)
SSH dead	+9 s	+14 s
Ping	dead	dead
PoE after halt, before cut	2.3 W	3.8 W
PoE adminEnable=2	0 W	0 W

How to tell the camera to shut down — options and M8

Anything that ends in edge-safe-shutdown.sh (or the same runc → sync → shutdown -h on-device) is a good trigger. PoE off is never the trigger.

Trigger	Where it lives	Fit
SSH / ops script	This Mac → jump → `edge-safe-shutdown.sh`	What we use in lab. Proven <10 s ghost-gone.
VMS / MQTT / proxy MCP	86.54 tells the edge “prepare halt”	Best fleet remote. Needs a small on-device listener (not built yet).
Watcher / reconcile	Desired-state “stopped”	Good for scheduled maintenance, not ignition-off.
M8 Button 1–3	On the camera. Luxonis `set_btn_callback` / GPIO IRQ	Best physical lab button. Hold 2 s → same halt. LED = shutting down. Buzzer optional.
M8 GPIO (ignition / ACC)	16× 3.3 V GPIO, ≤50 mA total; 2× 3.3 V 450 mA sensor rails	Vehicle key-off. Use an optocoupler — do not put 12 V on the pin. Falling edge = start halt.
M8 CAN / M8 CAN Adapter	SocketCAN 2.0A/B to 1 Mbps. Example ID `0x123` on button	Bus “ignition off” / J1939-style frame. Adapter is CAN-only ($99); exclusive with USB-C host and FSync splitter.
M8 RS232	1× serial to PLC / ECU	Fine if the vehicle already speaks RS232, not first choice.
M8 relays	4× SPDT latching 16 A / 400 VAC — outputs	Not an input. After halt they can drop accessory loads. Do not switch camera PoE with them.

M8 Controller Box (PR1, $249, powered from OAK4 M8 5 V): CAN, RS232, 16 GPIO, 4 relays, 3 buttons, 3 LEDs, 2× USB-A, buzzer. Examples run on the OAK4 via u2if — not host-driven. After shutdown -h the box dies with the camera, so it cannot cut Cisco PoE. Sequence: M8 trigger → software halt → VMS sees MQTT/SSH drop → Cisco adminEnable=2 (or ignition-switched midspan).

Recommended vehicle wiring: ignition ACC → optocoupler → GPIO; optional CAN frame as a second vote; Button 1 as a manual override. Same handler as the lab script. Strobe is only partially supported on PR1; ADC/I2C/SPI passthrough is not yet. On-device listener: scripts/m8-shutdown-listen.py (button-1 hold 2s / GPIO20 falling / CAN 0x100|0x123 data0=0x01). Copy onto the OAK after next boot; --dry-run first. Missing box exits M8_ABSENT.

Efficiency notes from run3

Simultaneous gi1+gi2 on is as clean as serial and ~half the wall time. Use serial only when isolating a unit.
After a clean halt, ping/SSH returns in ~37 s — faster than a mid-run hard cut story.
Long pole this run was first H.265 (~02:04), not RF-DETR (~01:00). T_nominal is rec_stable, not ghost.
Do not trust VMS mqtt_devices across a halt — names linger. Clock new files / new PIDs.
CBS wattage is a free boot oscilloscope. Detect=3 + 6–8 W (S1) / 8–13 W (S2) ≈ pipeline up.
S2 overlay 72% — prune keep-30 before the next long soak, not during boot.
Open .tmp count is the real “don’t cut yet” flag. Script first, watts last.
Enable-only (already 0 W) is cheaper than another 60/30 s drain when practicing boot.

Premium grades — last-known → nominal (run2)

Bar: proven 0W cut, same CID/layer/canary, RF-DETR with ghost_core, MQTT on 86.54, no reconcile. Premium time is ≤1 min; 3 min is lab-good.

Criterion	Premium means	Evidence	Grade
Last-known restore	Same CID, layer, not encode-only, no hollow, no rewrite	`layer1_rfdetr_thermal_safe`, ENCODE_ONLY=0, good CIDs, no reconcile	A
Config persist	Durable `/data/config` canary	token `poe-run2-20260814T014243Z` on marker + profile	A
RF-DETR resume	`T_rfdetr` after T=00:00	S1 01:01 · S2 00:59 (n_dets=0 allowed)	A−
Time to nominal	Prefer ≤1 min	S1 01:23 · S2 01:20 · 3 min bucket	B+
Cut hygiene	Mid-hold 0W + new PIDs	Both units after write-path fix	A
MQTT / VMS	Fresh on 86.54 only	shadow1+shadow2 grade 90	A−
Test craft	Serial, one port	Run2 after false-pass run0	A−
Overlay headroom	Auto prune before 95%	S1 31% / S2 52%; prune not automatic on cycle	B+
Preview JPEG	Out of pass bar	S2 :5001 can timeout while H.265 is fine	N/A

S1 recovery: A− · S2 recovery: A− · Session / ops craft: A− (B+ until write-path and password were fixed).

Latest outcome (run2) — last-known config + RF-DETR

Unit	PoE cut?	Config persisted?	T_rfdetr	T_nominal	Bucket
S1	Yes · 0W mid-hold · PIDs 1709/1888 → 1632/1835	Yes · token match	01:01	01:23	3 min
S2	Yes · 0W mid-hold · PIDs 1682/1820 → 1806/1913	Yes · token match	00:59	01:20	3 min

Both auto-returned last-known without reconcile. RF-DETR resumed with ghost_core (same poll). First H.265 appeared before ghost_core. MQTT on 86.54: shadow1+shadow2, grade 90.

Config settings under test (run2 canary)

Setting	S1	S2	After boot
Good oakapp CID	`aaf26107-e3d3-4a9d-9aff-a923d8884453`	`65acd4e6-58ee-4a38-ab37-e03e8ab9628c`	same CIDs running
Layer	`layer1_rfdetr_thermal_safe` · `ENCODE_ONLY=0`		unchanged
Camera profile (unchanged encode)	`v6_5fps_rf1_720p`	`transit_1080p_15fps_rf5`	left as-is
Canary token	`poe-run2-20260814T014243Z`		survived both boots
Marker file	`/data/config/poe-unit-marker.json`		token identical
`camera_profile.poe_unit_token`	same token written before drain		token identical
RF-DETR	on (not encode-only)		T_rfdetr ≈ T_ghost
MQTT / VMS	86.54:1883 / :8788 (not 86.46, not 87.x)		both shadows listed
Overlay at plant	31%	52%	no prune (under 80%)
S1 hollow CIDs	`1a8e8356…` / `3659a901…` never start		not started

Metrics — run2 vs run1

Metric	S1 run1	S1 run2	S2 run1	S2 run2
PoE mid-hold	0W / admin=2	0W / admin=2	0W / admin=2	0W / admin=2
T_ping / SSH	00:35	00:40	00:52 (down at 00:32)	00:59 ping / 00:37 ssh
T_ghost (both procs)	00:55	01:01	00:52	00:59
T_config (canary)	—	00:40	—	00:37
T_rfdetr	not gated	01:01	not gated	00:59
T_rec_start	00:35	00:40	00:52	00:37
T_rec_stable	01:15	01:23	01:34	01:20
T_mqtt (86.54)	~00:35	00:40	~00:52	00:37
T_nominal	01:15	01:23	01:34	01:20
Bucket	3 min	3 min	3 min	3 min
Reconcile needed?	no	no	no	no

T_rfdetr − T_ghost ≈ 0 on both. Model load is not the delay; ghost_core trails the agent by ~20s.

Run2 S1 timeline (unix T0=1786671857)

T+	Event	What happened
−60s	drain	gi1 adminEnable=2. mid-hold 0W / 0 mA.
00:00	power_on	adminEnable=1, still 0W — SOC coming up.
00:40	T_ping T_ssh T_layer T_config T_rec_start T_mqtt	Token `poe-run2-20260814T014243Z` present. ghost_agent only. MQTT lists shadow1.
01:01	T_ghost T_rfdetr	ghost_core PID 1835. ENCODE_ONLY=0 → RF-DETR path live.
01:23	T_rec_stable T_nominal	H.265 advancing. Last-known config + RF-DETR. 01:23 · 3 min.

Run2 S2 timeline (unix T0=1786671993)

T+	Event	What happened
−30s	drain	gi2 9.9W → 0W adminEnable=2. S1 left up (1835).
00:00	power_on	admin=1, 0W — booting.
00:37	T_ssh T_layer T_config T_rec_start T_mqtt	Token survived. ping still false. ghost_core not yet.
00:59	T_ping T_ghost T_rfdetr	Full pipeline + RF-DETR. New PIDs 1806/1913.
01:20	T_rec_stable T_nominal	01:20 · 3 min.

Run1 baseline (same day, no canary)

Unit	T_nominal	Notes
S1	01:15	T0=1786669300 · PIDs 2536/2586 → 1709/1888 · overlay 29%
S2	01:34	T0=1786669586 · PIDs 1780/1812 → 1682/1820 · overlay 48%

Run1 proved the CBS write path after false-passes (adminEnable=0 and expired-password read-only). Run2 added config persistence + RF-DETR clocks on that same path.

Issues found and mitigations

#	Issue	Mitigation
1	Cisco password expired → read-only session. Bastion login looked like “Bad User or Password”.	Operator reset. Writable login from VMS 86.54 (statusCode 0).
2	adminEnable=0 does not cut CBS PoE. Early cycles left PIDs alive (false-pass).	adminEnable 2=off, 1=on. Mid-hold 0W required before T=00:00.
3	POST XML without `action="set"` → “Wrong format of XML detected”.	POST `{PoEPSEInterfaceList}` with `action="set"`. In `cisco_switch_api.py`.
4	ghost_agent up ~20s before ghost_core.	Nominal waits for both. Run2 T_rfdetr tied to ghost_core.
5	Nested SSH truncates /health JSON.	Bastion `python3 urllib`: `True shadow1,shadow2 90`.
6	This Mac (87.37) has no L2 to 86.x.	Switch HTTP from 86.54; edges via jump; MQTT only on 86.54.
7	S1 overlay 100% (58GB H.265) blocked oakapp on the unplanned POE.	Run2 overlays 31%/52%. Reconcile prunes at ≥95%. Keep-60 pre-cycle if ≥80%.
8	Config might have been overlay-only (lost on reboot).	Canary on `/data/config` survived both run2 boots.

Top 10 — cut power-restored → nominal (after run2)

CBS write path action="set" + adminEnable 2 — proven run1 and run2.
Boot-pin good CID — both units auto-started; no reconcile.
Rotate H.265 before overlay ≥95% (not hit this run; still the 2026-08-13 killer).
LaunchAgent on 86.54: ping-up + no ghost_core by T+90s → runc. Scaffold: scripts/install-edge-reconcile-launchagent.sh.
NTP in parallel with oakapp — best-effort added to reconcile-edge-desired-state.sh.
Default start = non-TTY runc (oakctl hub password blocks).
Judge MQTT only on 192.168.86.54 — monitor does this now.
ghost_core trails agent ~20s (S1 00:40→01:01, S2 00:37→00:59) — don’t wait on preview JPEG.
go2rtc preview watchdog (S2 :5001 can timeout while H.265 is fine).
Pre-warm RF-DETR/DLC — not the bottleneck: T_rfdetr − T_ghost ≈ 0.

How to repeat

# Plant canary, then from VMS 86.54 after writable Cisco login:
# /data/config/poe-unit-marker.json + camera_profile.poe_unit_token
python3 /tmp/cisco-poe-cycle-lan.py gi1 60   # mid_hold must be 0W
python3 scripts/poe-unit-monitor.py --device S1 --t0 <unix> \
  --expect-token poe-run2-… --jsonl docs/grading/poe-unit-test-s1-s2-run2.jsonl
python3 /tmp/cisco-poe-cycle-lan.py gi2 30
python3 scripts/poe-unit-monitor.py --device S2 --t0 <unix> \
  --expect-token poe-run2-… --jsonl docs/grading/poe-unit-test-s1-s2-run2.jsonl

Raw: docs/grading/poe-unit-test-s1-s2.jsonl (run1) · docs/grading/poe-unit-test-s1-s2-run2.jsonl (run2) · docs/grading/poe-unit-test-s1-s2-run3.jsonl (run3 simultaneous) · docs/grading/poe-unit-test-s1-s2-run4.jsonl (run4 6:00am CDT one-shot). Detail twin: poe-unit-test-s1-s2-run2.html.

Safe shutdown (2026-08-14) — not a plug pull

Order: stop oakapp (close H.265) → sync → shutdown -h now → wait SSH/ping dead → then Cisco adminEnable=2. Script: scripts/edge-safe-shutdown.sh. VMS 86.54 left up.

Step	S1	S2
Pre: ghost PIDs	1632 / 1835	1806 / 1913
Pre: overlay	41%	72%
runc kill + delete	STOP_ISSUED	STOP_ISSUED
ghost_core gone	+4 s	+5 s
OS halt / SSH dead	+6 s	+9 s
Bastion ping	dead	dead
PoE after halt (before cut)	still ~6.1 W	still ~4.0 W
PoE adminEnable=2	0 W	0 W

Halt ≠ zero watts. Script allows 15–90 s for chunk close; this run closed in <10 s each. Serial both + PoE off ≈ 1 min. Do not PoE-cut while ghost_core is writing.

Catch-all 2026-08-14T20:34:38Z: S2 warn pre_event=202; mqtt storage_health=OK ignored (untrusted) · rc=0 PRUNED 172 KEPT 30 WAS 202 /dev/sda11 80761644 27331264 53413996 34% /overlay Warning: Permanently added 'ssh-remote.ghostprotocol.app' (ED25519) to the list of