Symphony management

Runs the Symphony 7.3.2 Advanced Edition management plane — EGO, SOAM, and (once Phase 5 lands) Host Factory — as a traditional, non-containerized install inside two RHEL 9 VMs: sym-master1 (primary) and sym-master2 (master candidate). This is the cluster brain every Windows compute host joins, and the component the whole supportability thesis leans on: management hosts run the exact IBM-documented install, on the exact IBM-documented OS family, with nothing repackaged.

Purpose

Both VMs are live and joined into a single EGO cluster (symgrid) as of Phase 2.1. Each VM’s cloud-init is rendered into a per-VM Secret and embeds the entitlement file and admin passwords — never committed as a literal manifest. symphony/config/ is the source of truth for the WAN-tuning settings; cloud-init’s apply-config.sh on each VM is a copy kept in sync by hand, not a live mount — both must be updated together.

Cluster identity is split across two credential classes, deliberately kept apart: the OS user egoadmin runs egoconfig subcommands, while EGO’s own built-in Admin identity is a separate login inside the grid’s own auth model. ego start (LIM/PEM/VEMKD) must run as root directly — these daemons start as root and drop privileges per job internally; a non-root LIM fails immediately.

Placement

Property Value

Cluster / Namespace

sym-aws / symphony

Network

Grid layer2 UDN (10.10.0.0/20), no NAT, no DNS for grid hostnames — /etc/hosts entries carry name resolution

Live addresses

sym-master1 = 10.10.0.31, sym-master2 = 10.10.0.34 (OVN’s allocator, not statically requested on this OCP version — re-run create-secrets.sh if a fresh cluster allocates differently)

Storage

gp3-csi (EBS, RWO), 60Gi each — not live-migratable; both masters are treated as pets that must be gracefully stopped before node-level maintenance that would otherwise trigger a KubeVirt eviction

Instance shape

u1.xlarge (4 vCPU/16Gi) — management hosts are not compute-bound at PoC scale

Burst exposure (Phase 3, not yet active)

Either a gated-out LoadBalancer Service, or a revisit to native-EC2 management hosts if the NLB’s SSM listener-count budget (~50) proves awkward (D8 revisit point)

Interfaces

Peer Direction Port / Protocol Purpose

installer-repo

outbound

HTTP 80

Fetch symeval-7.3.2.0_x86_64.bin, net-tools/libnsl RPMs

sym-master1 <→ sym-master2

bidirectional

7869 TCP+UDP (EGO_LIM_PORT, base)

LIM peer registration/heartbeat between primary and candidate

sym-master1 <→ sym-master2

bidirectional

7870 TCP (EGO_KD_PORT / vemkd)

Client entry point; candidate participates in cluster dispatch

sym-master1 <→ sym-master2

bidirectional

7871 TCP (EGO_PEM_PORT)

PEM inter-host resource reporting

Primary only

 — 

7872 TCP (egosc), 7873 TCP (repository service)

Service-package distribution, scheduler — not listened on the candidate (verified via ss)

Primary only

 — 

7874/7875 TCP (SD_ADMIN_PORT/SD_SDK_PORT)

Session Director admin/SDK

Windows compute hosts

inbound (join) / outbound (dial-back)

7869-7875 TCP + 7869 UDP

Compute host registers; master’s LIM dials back after registration — bidirectional firewall rules required on both sides, see Windows compute pool

Any grid host

as configured

SSM_SDK_ADDR 31000-31019, SSM_SIM_ADDR 31020-31039

Pinned SSM ranges — OS-random by default; pinned for firewall/NLB passability

Maturity

  • Idempotency guard trap (build-discovered, both masters, 2026-07-09): apply-config.sh’s original guard checked for `EGO_DYNAMIC_HOST_WAIT_TIME already present in ego.conf — but stock 7.3.2 ego.conf ships that key already defined at 60s, so the guard false-positived and the WAN-tuning fragment silently never applied on either master. Fixed by keying the guard on a fragment-unique setting (EGO_ENABLE_CHAN_KEEPALIVE) with an explicit sed-delete of the stock line first.

  • EGO_DYNAMIC_HOST_WAIT_TIME must never be deleted, only tuned: IBM’s ego.conf reference states verbatim that dynamic hosts cannot join at all if this key is undefined. EGO_DYNAMIC_HOST_TIMEOUT is the opposite trap: it defaults to not defined, meaning dead dynamic hosts are never auto-removed unless set explicitly (this fragment sets it to 15M).

  • /etc/hosts DNS-gap lesson: the grid UDN has no DNS for grid hostnames — VM guests get the cluster DNS server via DHCP but not the pod-style search domains, so the short-form installer-repo.symphony.svc fails to resolve from a VM guest even though it resolves fine from a regular pod; both cloud-init scripts fall back to the FQDN.

  • MTU pinned to 1500 as a build-proven fix, not a paper spec (IBM KB ibm16164145): the layer2 UDN hands out the pod-network jumbo MTU (8901 on the AWS underlay) via DHCP option 26; Linux masters honor it, Windows compute guests ignore it and stay at 1500. With masters at 8901, the ~3.8KB UDP LIM join payload left as a single jumbo frame the 1500-MTU Windows NIC silently dropped — hosts registered, then sat unavail with an empty guest-side LIM log. Both cloud-init templates now pin eth0 to 1500 before the Symphony install runs. Do not set spec.layer2.mtu on the live UserDefinedNetwork object — the UDN spec is immutable and it would wedge the Argo sync.

  • SSM port pinning is deferred to Phase 3 — only needed once a firewall/NAT sits between management and compute hosts (cross-site burst), not on the same-UDN Phase-2 topology.

  • Live-migration posture: gp3-csi is RWO — neither master can live-migrate, a real constraint for Phase-5 hardening not yet acted on in this build.

Verify

# 1. Both masters Running on the grid UDN with their expected addresses
oc get vmi -n symphony -l app=symphony-mgmt -o wide

# 2. EGO cluster is up and both hosts are members (run from an in-cluster pod with a
#    route to the grid UDN)
oc run egosh-check --rm -i --restart=Never --image=registry.access.redhat.com/ubi9/ubi-minimal -n symphony -- \
  bash -c 'command -v nc >/dev/null || microdnf install -y nmap-ncat; nc -zv 10.10.0.31 7869'

# 3. Full resource list via SSH from a pod that can reach the grid UDN
ssh egoadmin@10.10.0.31 'source /opt/ibm/spectrumcomputing/profile.platform && egosh resource list'

# 4. Confirm the WAN-tuning fragment actually landed (not a false-positive idempotency skip)
ssh egoadmin@10.10.0.31 'grep -E "EGO_ENABLE_CHAN_KEEPALIVE|EGO_DYNAMIC_HOST_WAIT_TIME|EGO_DYNAMIC_HOST_TIMEOUT" /opt/ibm/spectrumcomputing/kernel/conf/ego.conf'

# 5. Confirm live listener set matches the expected primary/candidate split
ssh egoadmin@10.10.0.31 'ss -ltnu | grep -E ":(786[9-9]|787[0-5])\b"'   # full 7869-7875 block + UDP 7869
ssh egoadmin@10.10.0.34 'ss -ltnu | grep -E ":(7869|7871)\b"'          # candidate: lim+pem only

Continue to Windows compute pool, the fleet that joins this master pair.