Network model

Symphony’s EGO registration is hostname/IP-anchored, not container-native. Every network decision in this design exists to give it what it needs: one flat layer2 network, no NAT, identical MTU everywhere, and firewall rules that allow the master to dial back to a host it just heard from. Three of these were real build lessons, not paper specs — this page states the invariant, then the failure that proved it.

The invariant

All Symphony hosts — management and compute, either site — sit on a single layer2 User Defined Network (UDN) with real, stable, non-NAT’d IPs. Symphony’s EGO registration breaks under masquerade NAT. Same base port 7869 on every host is mandatory.

MTU discipline — the ibm16164145 lesson

Masters and Windows guests must run the same MTU, pinned to 1500.

This was a real build lesson. The layer2 UDN’s DHCP hands out the pod-network jumbo MTU (8901 on the AWS underlay) via DHCP option 26. Linux masters honored it and ran at 8901; Windows guests ignore option 26 and stay at 1500. The master’s ~3.8 KB UDP LIM join payload then left as a single jumbo frame the 1500-MTU Windows NIC silently dropped — the compute host registered with the master, then sat unavail with an empty guest-side LIM log. That symptom matches IBM KB ibm16164145 exactly.

Symptom Compute host registers, then sits unavail with an empty guest-side LIM log — no error, just silence.

Root cause

Master eth0 at MTU 8901 (DHCP option 26 from the UDN), Windows guest NIC fixed at 1500. The master’s ~3.8 KB UDP LIM join payload leaves as one jumbo frame; the 1500-MTU Windows NIC drops it silently.

Fix

Pin masters' eth0 to MTU 1500 — persisted via nmcli and runtime ip link set — before Symphony install, so both sides match end-to-end.

Where it’s fixed

gitops/clusters/rosa/network-udn-grid.yaml, images/linux/cloud-init-master*.yaml.tmpl (git commit 5158077)

Keepalives and dynamic-host timing

EGO_ENABLE_CHAN_KEEPALIVE=y, EGO_CHAN_KEEPALIVE_TIME >= 180s (and below any firewall idle timeout) on the long-lived vemkd<→PEM / SSM<→SIM channels.

EGO_DYNAMIC_HOST_WAIT_TIME ships defined at 60s in stock ego.conf. IBM’s reference states dynamic hosts cannot join at all if the key is undefined, so config fragments must never delete it, only tune it. EGO_DYNAMIC_HOST_TIMEOUT defaults to not defined, meaning dead dynamic hosts are never auto-removed unless set explicitly.

A related idempotency trap was found in build: apply-config.sh’s original guard (`grep -q EGO_DYNAMIC_HOST_WAIT_TIME) false-positived against the stock 60s default already present in ego.conf, so the WAN-tuning fragment silently never applied on either master until the guard was changed to key on a fragment-unique setting (EGO_ENABLE_CHAN_KEEPALIVE) with an explicit sed-delete of the stock line first.

These values live in symphony/config/ego.conf.fragment, rolled onto hosts by apply-config.sh:

EGO_DYNAMIC_HOST_WAIT_TIME=120
EGO_DYNAMIC_HOST_TIMEOUT=15M
EGO_ENABLE_CHAN_KEEPALIVE=y
EGO_CHAN_KEEPALIVE_TIME=180

Bidirectional establishment

IBM does not document which side initiates every channel on the base-port range. The master’s LIM dials back to a newly-registered compute host — an inbound-only firewall rule on the compute side leaves it silently unavail even though outbound reachability tests pass clean. Allow bidirectional establishment on the 7869-7875 range and the pinned SSM ranges between management and compute hosts.

Design invariants — the full list

Every entry here cost real debugging time in this build; none of them is negotiable:

  • Same base port (7869) on every host — masters and compute, either site.

  • No NAT between grid hosts — layer2 UDN, real routable IPs, no masquerade.

  • Identical MTU across sites — pinned to 1500 on masters and Windows guests (IBM KB ibm16164145).

  • SSM ports pinned per symphony/config/sd-ssm-ports.xml, not left OS-random.

  • EGO_DYNAMIC_HOST_WAIT_TIME must be set (ships defined at 60s; never delete the key) or dynamic hosts silently fail to join.

  • Bidirectional 7869-7875 firewall establishment — inbound-only on the compute side leaves it silently unavail.

For the full port-by-port table, endpoints, and the live ss verification of what’s actually listening on the masters today, see Reference.

Continue to Reference.