UC2 — Burst to cloud

Backlog grows, a demand signal fires, and a new Windows compute host joins the grid from a golden image in minutes — not a procurement cycle. Two mechanisms drive this today: one built to prove the elasticity claim visibly (KEDA, designed for Phase 4), one built to run it for real (the Host Factory KubeVirt provider, the one custom component in this design, unbuilt, Phase 5).

The burst loop

uc2-burst-flow

The four sysprep steps, in the order they must run

Every clone runs the same specialize-pass script, and the order matters — it was built this way on purpose after real failures, not as a stylistic choice:

Step What Why this order

1. WinRM

Enable WinRM (Basic, unencrypted, port 5985) as the very first action

So a guest that hangs or crashes later in the script is still remotely inspectable — ADR 0012. Accepted only because the grid UDN is private, non-NAT’d, PoC-only.

2. Firewall

Open inbound EGO 7869-7875 via netsh advfirewall, not *-NetFirewallRule/WMI cmdlets

The WMI/CIM firewall provider is not functional this early in boot — every *-NetFirewallRule call fails with HRESULT 0x8004100a, fleet-confirmed in build. netsh talks to the firewall service directly.

3. Join

Rewrite EGO_MASTER_LIST="<master1> <master2>" directly in ego.conf

Windows egoconfig.exe has no join subcommand at all — given one it prints a paginated usage screen and blocks on stdin forever. Ports are already correct from the MSI’s BASEPORT=7869 property, so no other egoconfig call is needed.

4. Start LIM

Set the LIM service (that is its literal name, not EGO*) to Automatic, then start it — only after step 3

If LIM were Automatic from image build, it would auto-start before the master-list rewrite and exit fatally on the entitlement check — a host heading its own master list acts as a master and demands entitlement.

Once LIM starts with the corrected master list, the host registers, the master’s LIM dials back on 7869-7875 (inbound, not just outbound-reachable), and the host shows ok in egosh resource list.

Two return paths — deliberately not the same mechanism

KEDA demo (Phase 4, designed) Host Factory KubeVirt provider (Phase 5, unbuilt)

Status

Designed, not yet deployed — gitops/apps/keda/ holds only a README today

The one custom component in the entire architecture. Planned, not yet built — a genuine multi-person-month effort, not a weekend script.

Demand signal

Custom symphony-metrics-shim polling GET /platform/rest/symrest/v1/sessions → pendTasks over Symphony REST (:8180) — no Symphony Prometheus exporter exists, so this poller is new code

The standard symA requestor (throughput policy, tasks/min/slot) — Host Factory’s native demand signal, no custom polling needed

Scale mechanism

VirtualMachinePool’s standard Kubernetes `/scale subresource — no KubeVirt-specific KEDA integration required; mechanically proven elsewhere (Prometheus/RabbitMQ triggers)

IBM’s documented 5-script provider interface: getAvailableTemplates, requestMachines, requestReturnMachines, getRequestStatus, getReturnRequests

Scale-out

Same VirtualMachinePool clone + sysprep join path shown above

Same VirtualMachinePool clone + sysprep join path shown above, but VM creation is driven by the provider against an external kubeconfig (the burst-site cluster is not where Host Factory runs)

Scale-in

Not grid-aware. Deletes VMs without draining in-flight tasks, stranding work for rescheduling and leaving ghost unavail hosts in EGO. KubeVirt 1.6’s ordinal ScaleInStrategy (Ascending/Descending/Newest/Oldest/Random) only orders which VM dies first by creation time — never by idle/drain state. This gap is demonstrated on purpose in the demo script, not hidden.

requestReturnMachines — a negotiated, graceful return Host Factory drives itself, tuned by billing_interval (lazy return) against the burst site’s actual billing granularity

Prior art

Working example: KEDA scaling a VirtualMachinePool via Prometheus/RabbitMQ triggers

FINOS open-resource-broker (ex-Morgan Stanley/AWS, Apache-2.0) and google/symphony-gcp implement the identical 5-script contract against different substrates — neither has KubeVirt or Windows-VM code today

Decision record

ADR 0005

ADR 0005, ADR 0015

Both paths clone the same golden DataSource and run the same firstboot-symphony.ps1 join script — three different lifecycle managers (the static VirtualMachinePool, the KEDA demo, and the future Host Factory provider) converge on one image and one join mechanism. Running KEDA and the Host Factory provider against the same pool simultaneously would be a control-loop conflict — not a concern today since the provider does not exist yet, but a hard rule for when it does (ADR 0005).

Why the metal pool doesn’t scale with the VMs

Bare-metal workers are semi-static capacity, sized to the burst ceiling on a weekly/monthly planning horizon — node-level provisioning runs 5-30+ minutes (and on ROKS specifically, tens of minutes to hours once billing and cluster-join are counted). The VM is the burst unit: clone-to-AgentConnected was measured at ~5 minutes on real reclones in this build, roughly two orders of magnitude faster than a fresh metal join, on infrastructure that is already paid for (ADR 0015). This is why both burst mechanisms above scale VM replicas, never the metal pool itself.

Continue to KEDA metrics shim and Host Factory KubeVirt provider for per-component detail, or install phases for where each lands in the build sequence.