KEDA metrics shim

Drives Phase 4’s Windows compute burst demo: KEDA (packaged on OpenShift as the Custom Metrics Autoscaler) scales the sym-compute-win VirtualMachinePool up and down against real Symphony demand, proving VM-level elasticity end to end using entirely off-the-shelf mechanics — VirtualMachinePool’s standard Kubernetes `/scale subresource plus a small purpose-built poller (symphony-metrics-shim) that turns Symphony’s REST API into a KEDA-consumable metric. This is explicitly not the production burst mechanism — that is the Host Factory KubeVirt provider. ADR 0005 records the full rejection rationale.

Purpose

PLANNED, not yet deployed. gitops/apps/keda/ currently holds only a README describing the design; Phase 4 opens after the Phase 2/3 join and interconnect work closes out. When Phase 4 opens: the KEDA/Custom Metrics Autoscaler operator Subscription, the symphony-metrics-shim Deployment, and a ScaledObject targeting the ROKS VirtualMachinePool via a metrics-api trigger.

Placement

Property Value

Cluster

ROKS burst site (gitops/clusters/roks/ — does not exist yet, lands with Phase 3)

Namespace

symphony (co-located with the compute VMs it scales)

Scale target

pool.kubevirt.io/v1alpha1 VirtualMachinePool — the ROKS-side sibling of sym-compute-win

Demand source

Symphony REST API on the management host, :8180 (GET /platform/rest/symrest/v1/sessions → pendTasks) — reachable only over the grid UDN/interconnect, not exposed externally

Scaling bounds

min: 0 / max: = the ROKS metal pool’s VM capacity ceiling; cooldown tuned longer than the measured clone-to-AgentConnected time (~5 min) so the scaler doesn’t churn faster than a VM can actually come up

The shim authenticates to Symphony REST with credentials sourced from the same symphony-admin Secret pattern create-secrets.sh already populates for other consumers — no credential inline in a manifest or git. KEDA’s ScaledObject needs no elevated cluster permissions beyond patch on the /scale subresource of virtualmachinepools.pool.kubevirt.io in the symphony namespace — far narrower than the cluster-admin scope Argo’s application-controller holds.

Interfaces

Peer Direction Port / Protocol Purpose

Symphony REST (mgmt host)

outbound from shim

8180 TCP

Poll pendTasks/session demand — the metric KEDA scales on

symphony-admin Secret

in-cluster mount

 — 

REST auth credential, never inline in the manifest

KEDA metrics-api adapter

inbound to shim

HTTP (in-cluster Service)

KEDA queries the shim’s exposed metric value each polling interval

sym-compute-win (ROKS) VirtualMachinePool

outbound from KEDA

/scale subresource (K8s API)

ScaledObject.scaleTargetRef — the only KubeVirt-specific surface; KEDA has no KubeVirt integration beyond this generic subresource

ROKS Windows compute VMs

n/a (effect, not a shim interface)

 — 

Scale-out clones from the golden DataSource; scale-in deletes VMs — see Maturity

Maturity

  • DEMO-SCOPED, Phase 4, not yet deployed. This component exists to prove the elasticity claim visibly and cheaply, not to run production burst.

  • Scale-in is not grid-aware. KEDA/HPA-style scale-in deletes VMs without draining in-flight Symphony tasks, stranding work for rescheduling and leaving ghost unavail hosts in EGO — versus Host Factory’s requestReturnMachines, a negotiated, graceful return. This gap is demonstrated on purpose in the demo script (submit backlog → scale out → drain → cooldown scale-in → observe and document any task kills), not hidden.

  • KubeVirt 1.6 ordinal ScaleInStrategy caveat: VirtualMachinePool gained an optional ordinal selection policy (Ascending/Descending/Newest/Oldest/Random, default Random) in KubeVirt 1.6 — but this is ordinal/creation-time ordering only, never workload-state or idle/drain-aware. It can express "kill the newest VM first," never "kill the VM that’s actually idle." The core gap above still requires Host Factory to close in production.

  • Mechanically, KEDA-scaling-a-VirtualMachinePool is proven working elsewhere (Prometheus/RabbitMQ triggers against a VirtualMachinePool’s `/scale subresource) — not a novel integration, just a novel demand signal (the Symphony REST poller has no prior art; no Symphony Prometheus exporter exists).

  • The OpenShift-packaged Custom Metrics Autoscaler Operator reached GA at operator version 2.10.1, in the OCP 4.14 era, having run as Technology Preview through OCP 4.11-4.13 — a comparatively young operator lineage on this cluster’s 4.20 baseline, though its GA status by the time Phase 4 opens is not in question.

  • Two autoscalers must never point at the same VirtualMachinePool in production — once the Host Factory provider exists, running KEDA against the same pool would be a control-loop conflict. This demo’s pool is intentionally scoped to ROKS/Phase 4 only.

Verify

# 1. Confirm the shim is up and returning a plausible demand metric
oc get pods -n symphony -l app=symphony-metrics-shim
oc exec -n symphony deploy/symphony-metrics-shim -- curl -s localhost:8080/metrics

# 2. Confirm KEDA sees the ScaledObject as active and healthy
oc get scaledobject -n symphony
oc describe scaledobject symphony-compute-burst -n symphony

# 3. Submit a task backlog and watch the pool scale out, then join
egosh resource list          # from a Symphony client/mgmt host -- watch host count grow
oc get virtualmachinepool sym-compute-win -n symphony -w

# 4. After cooldown, confirm scale-in and record any stranded/killed tasks observed
egosh resource list          # hosts should drop; note any that went from 'ok' to gone mid-task

Continue to the Host Factory KubeVirt provider, the production burst mechanism this demo stands in for.