ADR 0012 — WinRM Baked into the Golden Image as the Debug Channel

virtctl guestfs (the libguestfs-tools-rhel9 appliance) cannot mount NTFS at runtime on this cluster — every mount attempt failed with "unsupported filesystem type" on both running and stopped guests, despite available-all-groups advertising ntfs3g as a compiled-in feature.

That left virtctl vnc screenshot and guestosinfo disk/CPU sampling as the only non-invasive diagnostics — enough to catch the InstallShield-wizard problem in ADR 0011, but useless for inspecting live service state, firewall rules, or config files, and it burned hours of blind screenshot-based debugging trying to work out why joined hosts sat unavail.

qemu-guest-agent’s `AgentConnected=True was repeatedly and wrongly treated as evidence the grid join had succeeded — it only proves the VM booted and the agent is running, nothing about EGO. This conflation had to be explicitly disciplined out of the debugging process (also recorded in Build traps #18).

Basic auth does not work — the WinRM server only offers Negotiate/NTLM even when Basic is explicitly enabled via WSMan:...\Auth\Basic; the handshake fails. NTLM against the local Administrator account works from the start.

Reachability testing must originate from an in-cluster pod on the grid UDN, not the operator’s own shell — the operator’s shell has no route to the private grid subnet (10.10.0.0/20), which produced a false-negative timeout against a known-good target and nearly got reported as a real WinRM failure before a throwaway probe pod proved the test itself, not the target, was invalid.

Enable WinRM (port 5985) as the very first action in firstboot-symphony.ps1, before any other network-dependent step — later steps (grid join, firewall rules, EGO service start) can then be diagnosed even if they fail.

Use NTLM authentication only (not Basic) against the local Administrator account.

Always test WinRM reachability from an in-cluster pod on the grid UDN, never from an operator’s own shell, to avoid false-negative route failures.

Restores a real command channel into Windows guests, replacing hours-long blind screenshot/guest-agent debugging with direct, scriptable inspection (service state, firewall rules, ego.conf contents, event logs) — this is what actually found the join root cause (documented in ADR 0014).

Enabling it first in firstboot means even a guest that fails every later specialize step is still diagnosable, rather than opaque.

Establishes a documented, repeatable test discipline (in-cluster probe pod, NTLM not Basic) that avoids re-discovering the same false-negative and auth-method traps on every future debugging session.

This is an open management port baked permanently into the golden image, not a debug-only, torn-down-after-use tool — it must be scoped to the private grid UDN only (no route from outside the cluster’s grid network) and treated as a standing attack surface, not a one-off convenience.

The WinRM credential (local Administrator password) must stay in lockstep with the autounattend.xml/sysprep-baked password and the corresponding Secret — the build already hit a version of this exact desync trap independently (the WinRM-auth-rejects-password issue suspected to be caused by create-secrets.sh regenerating WIN_ADMIN_PASSWORD out of sync with the already-rendered secret used by unattend.xml; open at time of writing). Any process that can silently rotate that password without re-baking the image breaks the debug channel along with everything else that depends on the same credential.