feat: dedicated GRE tunnel (Tunnel100) bypassing DZ-managed Tunnel500

Root cause: the doublezero-agent on mia-sw01 manages Tunnel500's ACL (SEC-USER-500-IN) and drops outbound gossip with src 137.239.194.65. The agent overwrites any custom ACL entries. Fix: create a separate GRE tunnel (Tunnel100) using mia-sw01's free LAN IP (209.42.167.137) as tunnel source. This tunnel goes over the ISP uplink, completely independent of the DZ overlay: - mia-sw01: Tunnel100 src 209.42.167.137, dst 186.233.184.235 - biscayne: gre-ashburn src 186.233.184.235, dst 209.42.167.137 - Link addresses: 169.254.100.0/31 Playbook changes: - ashburn-relay-mia-sw01: Tunnel100 + Loopback101 + SEC-VALIDATOR-100-IN - ashburn-relay-biscayne: gre-ashburn tunnel + updated policy routing - New template: ashburn-routing-ifup.sh.j2 for boot persistence Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 01:47:58 +00:00
1 parent 0b52fc99d7
commit 742e84e3b0
@@ -1,85 +1,61 @@
-# Bug: Ashburn Relay — 137.239.194.65 Not Routable from Public Internet
+# Bug: Ashburn Relay — Outbound Gossip Dropped by DZ Agent ACL

 ## Summary

 `--gossip-host 137.239.194.65` correctly advertises the Ashburn relay IP in
-ContactInfo for all sockets (gossip, TVU, repair, TPU). However, 137.239.194.65
-is a DoubleZero overlay IP (137.239.192.0/19, IS-IS only) that is NOT announced
-via BGP to the public internet. Public peers cannot route to it, so TVU shreds,
-repair requests, and TPU traffic never arrive at was-sw01.
+ContactInfo for all sockets (gossip, TVU, repair, TPU). The inbound path
+works end-to-end (proven with kelce UDP tests through every hop). However,
+outbound gossip from biscayne (src 137.239.194.65) is dropped by the
+DoubleZero agent's ACL on mia-sw01's Tunnel500, preventing ContactInfo from
+propagating to the cluster. Peers never learn our TVU address.

 ## Evidence

- Gossip traffic arrives on `doublezero0` interface:
+- Inbound path confirmed hop by hop (kelce → was-sw01 → mia-sw01 → Tunnel500
+  → biscayne doublezero0 → DNAT → kind bridge → kind node eth0):
  ```
-  doublezero0 In  IP 64.130.58.70.8001 > 137.239.194.65.8001: UDP, length 132
+  01:04:12.136633 IP 69.112.108.72.58856 > 172.20.0.2.9000: UDP, length 13
  ```
- Zero TVU/repair traffic arrives:
+- Outbound gossip leaves biscayne correctly (src 137.239.194.65:8001 on
+  doublezero0), enters mia-sw01 via Tunnel500, hits SEC-USER-500-IN ACL:
  ```
-  tcpdump -i doublezero0 'dst host 137.239.194.65 and udp and not port 8001'
-  0 packets captured
+  60 deny ip any any [match 26355968 packets, 0:00:02 ago]
  ```
- ContactInfo correctly advertises all sockets on 137.239.194.65:
-  ```json
-  {
-    "gossip": "137.239.194.65:8001",
-    "tvu": "137.239.194.65:9000",
-    "serveRepair": "137.239.194.65:9011",
-    "tpu": "137.239.194.65:9002"
-  }
-  ```
- Outbound gossip from biscayne exits via `doublezero0` with source
-  137.239.194.65 — SNAT and routing work correctly in the outbound direction.
+  The ACL only permits src 186.233.184.235 and 169.254.7.7 — not 137.239.194.65.
+- Validator not visible in public RPC getClusterNodes (gossip not propagating)
+- Validator sees 775 nodes vs 5,045 on public RPC

 ## Root Cause

-**137.239.194.0/24 is not routable from the public internet.** The prefix
-belongs to DoubleZero's overlay address space (137.239.192.0/19, Momentum
-Telecom, WHOIS OriginAS: empty). It is advertised only via IS-IS within the
-DoubleZero switch mesh. There is no eBGP session on was-sw01 to advertise it
-to the ISP — all BGP peers are iBGP AS 65342 (DoubleZero internal).
+The `doublezero-agent` daemon on mia-sw01 manages Tunnel500 and its ACL
+(SEC-USER-500-IN). The agent periodically reconciles the ACL to its expected
+state, overwriting any custom entries we add. We cannot modify the ACL
+without the agent reverting it.

-When the validator advertises `tvu: 137.239.194.65:9000` in ContactInfo,
-public internet peers attempt to send turbine shreds to that IP, but the
-packets have no route through the global BGP table to reach was-sw01. Only
-DoubleZero-connected peers could potentially reach it via the overlay.
+137.239.194.65 is from the was-sw01 LAN block (137.239.194.64/29), routed
+by the ISP to was-sw01 via the WAN link. It IS publicly routable (confirmed
+by kelce ping/UDP tests). The earlier hypothesis that it was unroutable was
+wrong — the IP reaches was-sw01, gets forwarded to mia-sw01 via backbone,
+and reaches biscayne through Tunnel500 (inbound ACL direction is fine).

-The old shred relay pipeline worked because it used `--public-tvu-address
-64.92.84.81:20000` — was-sw01's Et1/1 ISP uplink IP, which IS publicly
-routable. The `--gossip-host 137.239.194.65` approach advertises a
-DoubleZero-only IP for ALL sockets, making TVU/repair/TPU unreachable from
-non-DoubleZero peers.
+The problem is outbound only: the Tunnel500 ingress ACL (traffic FROM
+biscayne TO mia-sw01) drops src 137.239.194.65.

-The original hypothesis (ACL/PBR port filtering) was wrong. The tunnel and
-switch routing work correctly — the problem is upstream: traffic never arrives
-at was-sw01 in the first place.
+## Fix

-## Impact
+Create a dedicated GRE tunnel (Tunnel100) between biscayne and mia-sw01
+that bypasses the DZ-managed Tunnel500 entirely:

-The validator cannot receive turbine shreds or serve repair requests via the
-low-latency Ashburn path. It falls back to the Miami public IP (186.233.184.235)
-for all shred/repair traffic, negating the benefit of `--gossip-host`.
+- **mia-sw01 Tunnel100**: src 209.42.167.137 (free LAN IP), dst 186.233.184.235
+  (biscayne), link 169.254.100.0/31, ACL SEC-VALIDATOR-100-IN (we control)
+- **biscayne gre-ashburn**: src 186.233.184.235, dst 209.42.167.137,
+  link 169.254.100.1/31

-## Fix Options
+Traffic flow unchanged except the tunnel:
+- Inbound: was-sw01 → backbone → mia-sw01 → Tunnel100 → biscayne → DNAT → agave
+- Outbound: agave → SNAT 137.239.194.65 → Tunnel100 → mia-sw01 → backbone → was-sw01

-1. **Use 64.92.84.81 (was-sw01 Et1/1) for ContactInfo sockets.** This is the
-   publicly routable Ashburn IP. Requires `--gossip-host 64.92.84.81` (or
-   equivalent `--bind-address` config) and DNAT/forwarding on was-sw01 to relay
-   traffic through the backbone → mia-sw01 → Tunnel500 → biscayne. The old
-   `--public-tvu-address` pipeline used this IP successfully.
-
-2. **Get DoubleZero to announce 137.239.194.0/24 via eBGP to the ISP.** This
-   would make the current `--gossip-host 137.239.194.65` setup work, but
-   requires coordination with DoubleZero operations.
-
-3. **Hybrid approach**: Use 64.92.84.81 for public-facing sockets (TVU, repair,
-   TPU) and 137.239.194.65 for gossip (which works via DoubleZero overlay).
-   Requires agave to support per-protocol address binding, which it does not
-   (`--gossip-host` sets ALL sockets to the same IP).
-
-## Previous Workaround
-
-The old `--public-tvu-address` pipeline used socat + shred-unwrap.py to relay
-shreds from 64.92.84.81:20000 to the validator. That pipeline is not persistent
-across reboots and was superseded by the `--gossip-host` approach (which turned
-out to be broken for non-DoubleZero peers).
+See:
+- `playbooks/ashburn-relay-mia-sw01.yml` (Tunnel100 + ACL + routes)
+- `playbooks/ashburn-relay-biscayne.yml` (gre-ashburn + DNAT + SNAT + policy routing)
+- `playbooks/ashburn-relay-was-sw01.yml` (static route, unchanged)