Mirantis Releases Docker Enterprise 3.1

LEARN MORE

Today I Learned: How to Enable SSH With Keypair Login on Windows Server 2019

TL;DR – When deploying stuff automatically to Windows Server 2019, Windows Remote Management (winrm) is probably the preferred solution, with Remote Desktop (.rdp) for manual logins and fiddling. But you can also use SSH, which can be more convenient (or can at least feel more convenient) to Linux-native operators.

The goal here is to be able to open a terminal and type:

$ ssh -i my_private_key Administrator@<Windows Server 2019 URL or IP address>

… and instantly end up in PowerShell on the node, logged in as Administrator.

The catch is: end-to-end instructions for enabling and configuring SSH with keypair access on Windows Server 2019 are thin upon the ground. This tutorial pulls together gleanings from several recent blogs to show how this can be done.

As a bonus, it also shows how it can be scripted for injection into cloud-based Windows Server 2019 VMs at launch, causing these to come up SSH-accessible from the jump. This bypasses the need to:

  • Wait for the server to come up completely
  • Obtain and record the Administrator password (on AWS, this ironically involves presenting your private key in the ‘Connect’ dialog in order to decrypt your password)
  • Download the offered .rdp access file
  • Use the .rdp (with password) to log into the server

… before making the changes required so that you never need to do any of this again.

Why did I need to figure this out? Glad you asked. Over the past few weeks, in preparation for the launch of Docker Enterprise 3.1, we’ve been deploying a lot of demo clusters using daily builds and “Beta bits” (and bobs). A key feature in the 3.1 release, reflecting capabilities introduced with Kubernetes 1.17, is the ability to use Windows Server worker nodes in heterogeneous clusters.

The catch there: because the bits I was using weren’t yet fully-baked, our standard cluster-deployment tooling wasn’t yet ready to prep Windows workers (in fact, we’ve introduced new tooling in the form of Launchpad, which does this very well). Instead, I was using Ansible, running playbooks I’d already written to deploy Docker Enterprise manager nodes and Linux workers. But my Ansible couldn’t (yet) prep a Windows worker for me: installing Docker Engine – Enterprise and other bits. And if you know Ansible, you know that getting it to connect with Windows servers using winrc can be somewhat daunting for Linux-heads.

That (to quote this xkcd strip) broke my Linux-oriented workflow. Each time I deployed a new cluster, I had to leave my Linux operations VM, browse to EC2 on my Windows desktop, find the brand-new Windows Server instances I’d deployed on AWS as workers, decrypt and store their gobbledegook passwords via the arcane AWS Windows Server access dialogs, download the .rdp files, then navigate .rdp login repeatedly to visit and configure them, before I could join them to my cluster.

SSH-(with keypairs)-from-the-jump would make this easier. It would at least let me use SSH from my deployer VM command line, and potentially let me quickly extend my Ansible playbooks to do the Docker Engine – Enterprise installation magic automatically.

And after Googling for a while, I figured it out. Here’s how you enable SSH for keypair access on Windows Server 2019.

First, the manual approach:

Prerequisites:

  • An SSH keypair (private and public keys: id_rsa and id_rsa.pub). Your cloud service may have let you generate one or more of these, which you downloaded. If you need a new one, you can generate it from the Linux command line using ssh-keygen, following this DigitalOcean tutorial.
  • Windows Server 2019 running somewhere accessible.
  • Administrative credentials and a way of logging in (.rdp perhaps?) and opening PowerShell as the Administrator.
  • As needed, access to Security Groups administration, so you can add an Inbound rule opening up port 22 for SSH (perhaps enabling login only from your IP address, or only enabling connections from private subnet IPs, enabling SSH login via a VPN or jumpbox within your subnet).

And here’s the recipe, which (I swear) was working for me as of 5/6/2020 on standard Amazon EC2 Windows Server 2019 Full AMIs.

Once you’re at the PowerShell prompt, start by installing OpenSSH server. (Many recipes suggest also installing the OpenSSH client, but we’re not going to do that, here.)

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

Then configure the SSH server process to run at startup, and start it:

Set-Service -Name sshd -StartupType ‘Automatic’
Start-Service sshd

Having completed those steps, you should be able to log into your Windows Server 2019 machine using SSH, with passwords, as follows. Log in as the Administrator:

(from Linux) $ ssh Administrator@<Windows Server 2019 URL or IP address>

If you’re running on AWS, you’ll need to use your private key to decrypt your password from the Connect dialog in the EC2 console.

Now, just like Linux, we need to write our public key into a known address on the server, so that sshd can match it with the private key our SSH client will present when logging in. By default, sshd wants to see Administrator Group member keys (that’s you, Administrator) in a file called 

C:\ProgramData\ssh\administrators_authorized_keys

… which you will need to create.

As a Linux person, you might think of doing this with echo <keytext> > filename. But on PowerShell, that’s a bad idea, since operators like “>” use 16-bit Unicode encoding, which OpenSSH (sshd) can’t read. See this fascinating StackOverflow post about Windows and PowerShell character encoding norms, and recent updates in PowerShell Core.

In case you’ve already made this mistake (and trust me, I tried echo first), you’ll notice that an acceptably-encoded RSA public keyfile is about 400 bytes long, whereas the Unicode (unreadable) version is around 800. Just kill that “extended mix,” grab a new copy of your key, and paste it in the script as shown below, which uses the Set-Content commandlet (which outputs ANSI by default). Advanced users, non-US users, and others who may be dealing with cloud-based Windows Servers and PowerShell environments globally configured to use Unicode (for example) may still need to set an explicit encoding here. 

$key = "<paste private key here>"
$key | Set-Content C:\ProgramData\ssh\administrators_authorized_keys

Now, we need to modify the security settings on this file, so that:

  • inheritance is disabled
  • there are only two users: Administrator (you) and Administrators (the Administrators group)
  • and both users have Full Control

… or key-based access will not work. This is analogous to doing:

(in Linux) $ chmod 600 ~/.ssh authorized_keys

In Windows, we can do this manually by right-clicking on the file, selecting Properties, clicking the Security tab, pressing the button marked Disable Inheritance, manually removing users other than Administrator and Administrators, and ensuring that both Administrator and Administrators have Full Control permissions.

Or, we can do it programmatically in PowerShell, like this:

$acl = Get-Acl C:\ProgramData\ssh\administrators_authorized_keys # get access control list
$acl.SetAccessRuleProtection($true, $false) # eliminates inheritance
$acl.Access | %{$acl.RemoveAccessRule($_)} # delete all users and permissions
$administratorRule = New-Object # build a Full Control rule for Administrator system.security.accesscontrol.filesystemaccessrule("Administrator","FullControl","Allow")
$acl.SetAccessRule($administratorRule)
$administratorsRule = New-Object # ditto for Administrators system.security.accesscontrol.filesystemaccessrule("Administrators","FullControl","Allow")
$acl.SetAccessRule($administratorsRule)
(Get-Item 'C:\ProgramData\ssh\administrators_authorized_keys').SetAccessControl($acl)

The last line imposes the modified acl back on the file. Using SetAccessControl avoids a small bug in Set-Acl, which seems to have caused some pain as well.

As a last step, we can (optionally) set the default login shell of the Administrator to PowerShell, instead of the old-school command shell.

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

Finally, we restart the sshd server:

restart service sshd

At this point, you should be able to log into your Windows Server 2019 box from any SSH, using your private key, as shown up top, and end up in PowerShell. If you want to use PuTTY, you’ll probably need to convert your private key to RSA format with PuttyGen, as described in this recipe.

If you’re using cloud-based Windows Server VMs, this script can also (in most cases) run as a User Data script, when you launch a new box. In AWS EC2, for example, you can just pick up the below script (along with the required <powershell></powershell> tags) and drop it right into the EC2 webUI when launching a new instance from the EC2 console, or copied into a file invoked by an AWS CLI run-instances command. 

<powershell>
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Set-Service -Name sshd -StartupType ‘Automatic’
Start-Service sshd
$key = "<paste public key here>"
$key | Set-Content C:\ProgramData\ssh\administrators_authorized_keys
$acl = Get-Acl C:\ProgramData\ssh\administrators_authorized_keys
$acl.SetAccessRuleProtection($true, $false)
$acl.Access | %{$acl.RemoveAccessRule($_)} # strip everything
$administratorRule = New-Object system.security.accesscontrol.filesystemaccessrule("Administrator","FullControl","Allow")
$acl.SetAccessRule($administratorRule)
$administratorsRule = New-Object system.security.accesscontrol.filesystemaccessrule("Administrators","FullControl","Allow")
$acl.SetAccessRule($administratorsRule)
(Get-Item 'C:\ProgramData\ssh\administrators_authorized_keys').SetAccessControl($acl)
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force
restart-service sshd
</powershell>

Let the instance spin up, and you should be able to log right into it from your terminal, via SSH and your private key.

John Jainschigg

Today I Learned (TIL) is an intermittent journal of somewhat half-baked solutions to “speed bump” problems encountered by Mirantis Technical Marketing folks working well beyond our native spheres of expertise for research purposes. None of what we write about here has been checked by grown-ups, or is appropriate for production without further validation. Use at own risk. Comments, cautions, and suggestions for improvement are very welcome! Email jjainschigg@mirantis.com (Twitter: @jjainschigg).

WEBINAR RECORDING
What's New in Kubernetes 1.18
WATCH NOW