Host Factory KubeVirt provider

This is the one custom component in the entire architecture. Every other layer — OpenShift, OpenShift Virtualization, OpenShift GitOps, OpenShift Pipelines, Windows Server, Symphony itself — is somebody’s supported product. The Host Factory KubeVirt provider is the exception: a purpose-built implementation of IBM Spectrum Symphony’s documented 5-script Host Factory provider interface, targeting OpenShift Virtualization/KubeVirt as the burst substrate, because no built-in Host Factory provider does this. It is what lets Symphony’s own demand-driven burst mechanism — the symA requestor (throughput policy, tasks/min/slot) — provision and return Windows compute VMs on OpenShift, the same way it already provisions EC2 instances or IBM Cloud VSIs through IBM’s stock providers.

Purpose

PLANNED, Phase 5. Not yet built. This is the design gap this reference architecture is honest about: the Phase 4 KEDA demo proves VM-level elasticity mechanically, but it is not grid-aware and is explicitly scoped to the demo, not production. When Phase 5 opens: five scripts implementing IBM’s Host Factory provider contract, configured under $HF_CONFDIR/providers/<name>/ on the Symphony management hosts, plus whatever watcher/reconciliation logic the provider needs to track VM state against Host Factory’s expected machine lifecycle.

Effort signal: multi-person-month, not a weekend script. External auth, VM lifecycle-to-Host-Factory-state mapping, sysprep injection, and lazy-return tuning are each nontrivial pieces.

Prior art — same contract, different substrate, not a from-scratch interface design:

  • FINOS open-resource-broker (ORB) — Apache-2.0, "Incubating" under FINOS governance, contributed jointly by Morgan Stanley and AWS. Implements the identical 5-script Host Factory interface against pluggable Kubernetes (Pod/Deployment/ StatefulSet/Job) and AWS (EC2 RunInstances/EC2Fleet/SpotFleet/ASG) backends. Correction to how this is often described: ORB is not a documented "successor" to morganstanley/symphony-hostfactory — FINOS describes it as extending the ecosystem.

  • google/symphony-gcp — actively maintained, independently implements the same 5-script contract, targeting both GCE VMs and GKE pods via a CLI provider + Kubernetes operator with CRDs — the closest architectural analog to a KubeVirt provider (VM backend + Kubernetes-native control loop in one project).

  • Neither analog has KubeVirt or Windows-VM code today — this provider would be new ground on both axes, not a port of an existing implementation.

Implementation strategy (2026-07-11): ORB-first. The provider will be built as a KubeVirt backend module inside FINOS ORB — reusing ORB’s hardened 5-script/watcher framework instead of re-implementing it, and making the work upstream-able to FINOS — with a standalone script implementation as the fallback, not the plan.

The ORB route also inherits an option lane this design notes but does not commit to: ORB’s existing Kubernetes module can provision Linux SOA capacity as native pods under the same Host Factory demand signal, should the grid ever want cheap Linux compute alongside the Windows VM fleet. Caveats that keep it an option: not an IBM-supported production pattern; pods must still satisfy the grid network invariants (a secondary attachment onto the grid UDN — no NAT, same base port); the management plane stays on RHEL VMs (ADR 0008) and Windows compute stays full VMs (ADR 0001) — the archived IBM Symphony operator (CE-only, 64-core cap, impersonation disabled) is not the route back to pods.

This lane is now empirically refined, not paper (Track C C0 research, 2026-07-12). The archived operator is confirmed dead on OCP 4.20, not just archived: the repo has no CRD/RBAC/OLM assets in any of its 30 commits (client Template + README prose only), and the real certified bundle is credential-gated and absent from OperatorHub on the live sym-aws cluster. Separately, both the operator and the underlying ibmcom/spectrum-symphony CE image hardcode the base port to the 17869 family at image-build time, with no CRD field or env var to reach this grid’s invariant 7869 — the operator itself cannot serve 7869 without probe-driven crash loops. Verdict for the related containerized-master experiment (a separate track from this Linux-pods lane): GO-WITH-SUBSTITUTION. Drop the operator entirely and hand-write a plain StatefulSet running the CE image directly, own liveness/readiness probes, egoconfig setbaseport 7869 -f in a pre-start hook — untested but plausible and product-supported, gated on confirming LIM/PEM/RS/SD actually come up on 7869 before any Windows KubeVirt join is attempted. This does not change the verdict on this Linux-pods option lane itself (still an option, not a plan) — it only replaces "the archived operator" with "a hand-written StatefulSet on the CE image" as the credible substrate wherever a containerized Symphony process is being considered. See docs/plans/burst-plan.md Track C and docs/HANDOFF-operator-hybrid.md in the source repo.

Placement

Property Value

Provider scripts

Live on the Symphony management hosts (RHEL VMs), under $HF_CONFDIR/providers/<provider-name>/ — where Host Factory’s hostProviders.json points it to look

Where VMs get created

The burst-site cluster (ROKS, Phase 3) — a different cluster from where the scripts run, which is why authentication cannot be in-cluster

Network

The burst site’s localnet UDN/VNI, the same no-NAT, real-IP grid network every other Symphony host in this design uses — a burst VM must be reachable from the AWS-side masters exactly like any other compute host

Image

The same golden DataSource (detail) the KEDA demo and the static sym-compute-win VirtualMachinePool both use — one image, three different lifecycle managers

External kubeconfig authentication is required, not optional. IBM’s built-in openshiftprov Host Factory provider only supports in-cluster authentication — usable when Host Factory itself runs inside the target OpenShift cluster. Here, Host Factory runs on the AWS-side management hosts and must provision VMs on a separate burst-site cluster, so openshiftprov’s auth model does not fit; this is genuinely new code, not a configuration change. The kubeconfig/service-account token is a credential like any other — created as an OpenShift Secret, never committed to git. The service account should be scoped to create/delete/get on `VirtualMachine and DataVolume objects in the symphony namespace only, not cluster-admin — narrower than the scope Argo’s application-controller holds on the management cluster.

Lifecycle mapping is the trust boundary. The provider’s getRequestStatus and getReturnRequests responses are what Host Factory trusts to know whether a machine is really up, really returned, or still billing — an incorrect mapping (e.g., reporting a VM "complete" before its EGO join finishes) would let Symphony route work to a host that isn’t actually joined yet.

Interfaces

Peer Direction Port / Protocol Purpose

Host Factory core (hfcore, on mgmt host)

invokes provider

local exec, <script>.sh -f input.json

The 5-script contract: getAvailableTemplates, requestMachines, requestReturnMachines, getRequestStatus, getReturnRequests (optional)

Provider process

reads config

$HF_CONFDIR/providers/<name>/ + HF_PROVIDER_* env vars

Template definitions, target-cluster kubeconfig reference, provider-specific tuning

Burst-site OpenShift API

outbound from provider

443 HTTPS (external kubeconfig)

Create/delete VirtualMachine/DataVolume objects — NOT in-cluster auth, unlike the built-in openshiftprov

Golden DataSource (burst cluster)

referenced by created VMs

CDI clone

Source image for every provider-provisioned VM

Burst-site localnet UDN/VNI

VM network attachment

Symphony port matrix (base 7869 block)

No-NAT, real-IP reachability to the AWS-side masters — same invariant as every other grid host

symA requestor (on mgmt host)

drives demand

internal EGO/Host Factory protocol

The standard Symphony demand signal this provider serves — no custom demand logic needed, unlike the KEDA demo’s SYMREST poller

Maturity

  • PLANNED, Phase 5. Not yet built. This is the honest design gap the reference architecture states plainly rather than glossing over — the whole supportability thesis rests on it being the only such gap.

  • Per-VM sysprep injection must carry the same join secrets as the static pool — the provider must inject hostname, $EGO_CONFDIR/hosts entries for the master(s), and the EGO_MASTER_LIST rewrite the same way firstboot-symphony.ps1 does for statically-pooled VMs, since Windows egoconfig.exe has no join subcommand at all (git commit 5158077). Whatever credential material this requires (win-admin password, entitlement) must come from the same Secret-based flow create-secrets.sh already establishes, not be duplicated into provider config files.

  • Being "the one custom component" also means it is the one component with no vendor to call at 3am — its correctness is this project’s responsibility alone, which is precisely the trade-off the supportability thesis names explicitly rather than hides.

Verify

Not yet buildable (Phase 5), so verification today is a contract test design, not a running check — the plan for closing this gap when Phase 5 opens:

# 1. Each of the 5 scripts must be independently invocable exactly as Host Factory calls
#    them -- <script>.sh -f input.json -- and produce Host-Factory-schema-valid JSON on
#    stdout. Contract test: feed each script a canned input.json fixture, assert output
#    matches the IBM-documented response schema (status, machines[]/requests[] shape).
./getAvailableTemplates.sh -f test/fixtures/available-templates-input.json | jq .
./requestMachines.sh -f test/fixtures/request-machines-input.json | jq .

# 2. State-machine test: requestMachines -> poll getRequestStatus until "complete" ->
#    assert the reported machine is actually EGO-joined (egosh resource list shows
#    'ok'), not just VM-Running -- this is the trust-boundary check from Placement.
egosh resource list | grep <hostname-from-requestMachines-response>

# 3. Teardown test: requestReturnMachines -> poll getReturnRequests (or
#    getRequestStatus) until the VM is gone -> assert EGO no longer lists the host and no
#    ghost 'unavail' entry remains.
egosh resource list | grep <hostname>   # expect: no match

# 4. Auth-boundary test: run all of the above with the provider's external kubeconfig
#    scoped to the burst namespace only -- confirm no operation succeeds outside that
#    scope.

Continue to Interconnect, the cross-cloud network path this provider’s provisioned VMs must reach the masters over.