Part#2: Removing SSH 'pem' files from Jump Boxes in AWS - Getting it working

Hi Everyone,

In Part #1 of this series : Removing SSH 'pem' files from Jump Boxes in AWS - Introduction, I described the problem of SSH management and started on a solution to restrict the spread of SSH key distribution across the servers in an environment.

In this post, we'll put this all together and show how the AWS Parameter Store can help us manage our SSH keys.

Uploading SSH keys to the AWS Parameter Store

As mentioned in the first document, it's not possible to just copy/paste SSH 'pem' keys (private keys) via the AWS Console - for some reason it translates the EOL characters into spaces. Instead, we'll need to do this via the command line - so let's see how to do this.

First, we need to change the IAM Role for the Instance where we will be running the command - to allow the Instance to 'put' values into the AWS Parameter Store.
In this case, I altered the non-prod jumpbox and gave it the : RxR-NonProd-Parameter-Keystore-Manager' Role (but will change back to Read Only after the put is done).
Please see the first document on how this is done.

#!/bin/bash

echo 'Value :' $1

printf "#!/bin/bash\n" > insert-pem.sh
printf "aws ssm put-parameter --name "/example/$1" --type "SecureString" -- 
overwrite --region ap-southeast-2 --value " >> insert-pem.sh
printf '"' >> insert-pem.sh
cat $1 >> insert-pem.sh
printf '"\n' >> insert-pem.sh
chmod +x insert-pem.sh

If we now run this, using the pem file 'example.pem', a new file - insert-pem.sh, will be generated :

cat insert-pem.sh
#!/bin/bash
aws ssm put-parameter --name /example/example.pem --type SecureString --overwrite 
--region ap-southeast-2 --value "-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAuuWiGeI5o0XWTiG3wIpKpHl9bsnHy62vq5prz+sF+LERu0wj
7dwpvjMYw4Py1SdiaMpAuNDyb3m8Te6knf/1k6PNKBdzOp7fwNwaKC3Xxv9A8aNJ
FcWiSGYnS6Ym+y8L3AulrXuVSi1hcdnM7lWJL8rlP1PLa/RUqkJVZOFPS/AURH1H
2ouinba91a5UlT/91IeaX6rkPlMrESyS9jJ0MXvQDAF5CeN7EfgeqW9q3BOh937G
p/VjH53kImSd37c3rKPoAPmRwUGW2wsJv7Aj3s++ob12VYofV2cTRxDKd/C/w+bm
+4dwEV0UA5Y5DSBZIhg6aQhY/570HNLKBRSOzwIDAQABAoH/fNbmcGOsHGOJwqbD
FtlJApQkNNyTmHlr3jRHz5otYvDj5zf0x+1H9MQsxmxcyASGt3hhwxHO3qdyIJxH
WUKfxv6jCySe04lL7IcY8b2tZD8cIkvvhZt9yt5xYRaa8y15voIJKNhj4Chv7pyN
RqXb/RCKEMLoHjQWWaP1Zm0Zip6udkJ+U9/p1CsBs7W6paggnvTcHFir/j7LK8fb
YPKyTTzn/5YKudkzyeadnqEm1/yrBNDrYsCcBcEjECaGiJWFY71tgFZbgx/R2qlw
IE5UAqTD0tseiZLiSkwK5wwfA79vNg8Y3VqJak3n0LRnV+bmLpqvJ2Ti2jWreel0
3iApAoGBAOO+5OIIXHsjKDEEbsE6+08stl2DthAXbJbJeFyx1Y8sV/Q0JptCwSHK
3m+CsEQEJibBCd8udCbgva2zpFOCpDZd3xWd9fcVJTHIv+PFkRU4HLFqu4kq5CJN
Migaeya8jczIhOCIUteRjwjTYktUp7izqPTbQZLGVNUxoQC6x42NAoGBANIVZw+b
vQP+LXyLT2Xn6bK7J1kz+aU7+9uEOWjt0w0RNwJp1NFVhUNdZu0/1HP7plpEs9+v
5AytHVqit+yFVMtzzhPlnIRxAPNTcaKNN9PKq+cWo7IxGKHr/oaSQEVSyZM7BL60
ZuUyMd4Y1Uj4Y033aXvPh9ZSmK8buh70GJDLAoGALxVvIl/SJs785y1gbGhyPksj
JLAvOqJtG7tpO9i0KxZi1KTXem/Rl9fmJq2hHV6yOH7eOJrU/pil/jxOc8llg6vi
jAiNKljfyKwVJ9TXish//hnvzYrlgUXuYaYf4cFHm5e1COqGHfUE2jFBeAt3ZVMX
Ug/mPZ7tOOo6u/jhep0CgYEAiCdK1fjPNYlEHZWD7b2x3h4hUFS5FPedMRWux2xq
/esMRcolHU5ZweEE1foqTMjpdVNVaEFsgTKo9ZbDsMxwujLhek+zcrpyLUEFQp5X
cjHF20z25oOfPUHfgYPDl6oh7gOv/ZCZpy0y2IgrotEoOt2ARbeoh6ScQ9CqfWkh
iqUCgYEA2CFerVstBPJcjIV7ZWXtrpjfsyTGTm0YVOMtlZOSch3jYRcOFBRL9MR1
edwKSSNaW1o/NUjuHZiQyzJljPUPdbuoa+VuiFrs5SxDGGckA9wpkmp6ihwIHm3B
lZj8LMMa6l4WRVI7jpjHnsF8Kk756VV6eLfPGQ8gVMP9/kWZYWc=
-----END RSA PRIVATE KEY-----
"

Let's run this and check that it has inserted correctly :

Oops! this is what happens if you don't set the 'Manager' version of the IAM Role (ie. still using 'Read-Only' :

[ec2-user@jumpbox.nonprod ~]$ ./insert-pem.sh

An error occurred (AccessDeniedException) when calling the PutParameter operation: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/RxR-NonProd-Read-Parameter-Store/i-xxxxxxxxxxxxxx is not authorized to perform: ssm:PutParameter on resource: arn:aws:ssm:xxxxxxxxxxxxxxxxx:parameter/example/example.pem
[ec2-user@jumpbox.nonprod ~]$

After setting the correct Role, let's try that again!

[ec2-user@jumpbox.nonprod ~]$ ./insert-pem.sh

All good this time. And we can see it in the Parameter Store too!

An interesting note here - thanks to my colleague Craig Barr.
Craig pointed out to me that it's possible to even dispense with the 'insert-pem.sh' file completely - due to a really neat trick that I wasn't aware of with the '--values' clause.
The whole 'insert-pem/sh' file can be replaced with a one liner!

aws ssm put-parameter --name /example/example.pem --type SecureString --overwrite --region ap-southeast-2 --value "`cat privateKey`"

Well, that blew me away - how cool allowing back ticks to execute 'bash' commands - neat! Thanks Craig!

Now let's retrieve it to make sure that it comes back with Carriage Return intact :

[ec2-user@jumpbox.nonprod ~]$ aws ssm get-parameters --name /example/example.pem --with-decryption --query "Parameters[*].{Value:Value}" --region ap-southeast-2 --output text
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAuuWiGeI5o0XWTiG3wIpKpHl9bsnHy62vq5prz+sF+LERu0wj
7dwpvjMYw4Py1SdiaMpAuNDyb3m8Te6knf/1k6PNKBdzOp7fwNwaKC3Xxv9A8aNJ
FcWiSGYnS6Ym+y8L3AulrXuVSi1hcdnM7lWJL8rlP1PLa/RUqkJVZOFPS/AURH1H
2ouinba91a5UlT/91IeaX6rkPlMrESyS9jJ0MXvQDAF5CeN7EfgeqW9q3BOh937G
p/VjH53kImSd37c3rKPoAPmRwUGW2wsJv7Aj3s++ob12VYofV2cTRxDKd/C/w+bm
+4dwEV0UA5Y5DSBZIhg6aQhY/570HNLKBRSOzwIDAQABAoH/fNbmcGOsHGOJwqbD
FtlJApQkNNyTmHlr3jRHz5otYvDj5zf0x+1H9MQsxmxcyASGt3hhwxHO3qdyIJxH
WUKfxv6jCySe04lL7IcY8b2tZD8cIkvvhZt9yt5xYRaa8y15voIJKNhj4Chv7pyN
RqXb/RCKEMLoHjQWWaP1Zm0Zip6udkJ+U9/p1CsBs7W6paggnvTcHFir/j7LK8fb
YPKyTTzn/5YKudkzyeadnqEm1/yrBNDrYsCcBcEjECaGiJWFY71tgFZbgx/R2qlw
IE5UAqTD0tseiZLiSkwK5wwfA79vNg8Y3VqJak3n0LRnV+bmLpqvJ2Ti2jWreel0
3iApAoGBAOO+5OIIXHsjKDEEbsE6+08stl2DthAXbJbJeFyx1Y8sV/Q0JptCwSHK
3m+CsEQEJibBCd8udCbgva2zpFOCpDZd3xWd9fcVJTHIv+PFkRU4HLFqu4kq5CJN
Migaeya8jczIhOCIUteRjwjTYktUp7izqPTbQZLGVNUxoQC6x42NAoGBANIVZw+b
vQP+LXyLT2Xn6bK7J1kz+aU7+9uEOWjt0w0RNwJp1NFVhUNdZu0/1HP7plpEs9+v
5AytHVqit+yFVMtzzhPlnIRxAPNTcaKNN9PKq+cWo7IxGKHr/oaSQEVSyZM7BL60
ZuUyMd4Y1Uj4Y033aXvPh9ZSmK8buh70GJDLAoGALxVvIl/SJs785y1gbGhyPksj
JLAvOqJtG7tpO9i0KxZi1KTXem/Rl9fmJq2hHV6yOH7eOJrU/pil/jxOc8llg6vi
jAiNKljfyKwVJ9TXish//hnvzYrlgUXuYaYf4cFHm5e1COqGHfUE2jFBeAt3ZVMX
Ug/mPZ7tOOo6u/jhep0CgYEAiCdK1fjPNYlEHZWD7b2x3h4hUFS5FPedMRWux2xq
/esMRcolHU5ZweEE1foqTMjpdVNVaEFsgTKo9ZbDsMxwujLhek+zcrpyLUEFQp5X
cjHF20z25oOfPUHfgYPDl6oh7gOv/ZCZpy0y2IgrotEoOt2ARbeoh6ScQ9CqfWkh
iqUCgYEA2CFerVstBPJcjIV7ZWXtrpjfsyTGTm0YVOMtlZOSch3jYRcOFBRL9MR1
edwKSSNaW1o/NUjuHZiQyzJljPUPdbuoa+VuiFrs5SxDGGckA9wpkmp6ihwIHm3B
lZj8LMMa6l4WRVI7jpjHnsF8Kk756VV6eLfPGQ8gVMP9/kWZYWc=
-----END RSA PRIVATE KEY-----
"

Yes - That works! Great. So we know that we can extract the key with just a command line.

This is where things get tricky!

This is a good place to stop for a second and think about where we're at. We know that we can retrieve an SSH 'pem' file from the AWS Parameter Store. That's great!

But thinking about how we do connections via the execution of a SSH command - SSH wants a file that it can read with its '-i' parameter. It also wants that file to be 'chmod 600' - restricted user read only permissions.
This leads to an interesting conundrum, since we don't want to store, or leave behind, any 'pem' files or scripts on the jump boxes.

So what we want to do is :

  • Somehow call a script, that can't exist on the jump box, that will extract the target SSH 'pem' file to the jump box and save it as a file
  • SSH uses that file to log in (using ssh -i <pem-file>)
  • That 'pem' file is then deleted as soon as possible - but the SSH connection must remain.
  • Then the script itself must not be left behind on the jump box.

The answer to all this is to go one step further with the AWS Parameter Store, and use it to also store the script we'll be needing!
Then, from the MobaXterm command line, we can execute a command that extracts the script from the AWS Parameter Store, which in turn sets up the SSH connection. As part of it's execution, the script ensures that the downloaded SSH 'pem' key, a well as itself, are deleted!

So, to do this, we'll take the same approach as before when uploading an SSH 'pem' key - but this time upload our script instead.
The script looks like this :

[ec2-user]# cat connecter.sh
#!/bin/bash

#
# Script to connect a user to a server.
#
# A. Nakon. 02/2018.
#
# To run this script : ./connecter <pem file name>  <target IP address>
#
#  eg.  ./connecter <pem-file> <ip-address>
#

export MYFILE=$(date +'FN%s%N')
aws ssm get-parameters --name /example/${1} --with-decryption --query "Parameters[*].{Value:Value}" --region ap-southeast-2 --output text > $MYFILE
chmod 600 $MYFILE
(sleep 1 && rm -f $MYFILE) &
ssh -i $MYFILE ec2-user@${2}
unset MYFILE

Note on the line "(sleep 1 && rm -f $MYFILE) &" - the sub-shell command to remove the SSH 'pem' file one after a (1) second sleep. I needed to take this approach since attempting to post-pend an 'rm' command after the SSH command would not work. That is, the 'pem' file would hang around on the jump box until the SSH command exited - not what we want (since any interim login to the jump box would be able to see the SSH pem file!).

So, that takes care of the SSH execution and the removal of the 'pem' file - but how about the script itself - since we need to also remove it from the jump box after it has done it's job.

The answer is to take the same approach - but this time apply the 'rm' within the execute string we send through to MobaXterm :

Moba SSH command

The actual execute string looks like this :

aws ssm get-parameters --name /example/connecter.sh --query "Parameters[*].{Value:Value}" --region ap-southeast-2 --out text > connecter.sh && chmod +x connecter.sh ; (sleep 2 && rm connecter.sh &);  ./connecter.sh example.pem 192.168.0.10

Breaking this down ...

  • We call AWS to extract the script 'connecter.sh', saving the extracted file as 'connecter.sh' and making it executable.
  • We then create a sub-shell command that will sleep for 1 second before deleting the script and make it a background task.
  • Without pause, we run the 'connecter.sh' script with two (2) parameters - the 'pem' file we'll be needing plus the target server IP Address.

And that's it!

When we connect using this methodology with MobaXterm, we will SSH connect to our target server and there will be no 'pem' or script files left over on the jump box - exactly what we want!

Reviewing it all ...

It's a good idea now to review what we have achieved with all this :

  1. We have vastly reduced the set of 'pem' keys we need to store on the servers in the environment.
    Remembering that previously we had two options - (1) Store SSH 'pem' files on the jump boxes, or (2) Store the full set of SSH 'pem' files on the Windows server where MobaXterm runs (using the network settings / SSH gateway option) -
    we have now reduced all this to just two (2) SSH 'pem' files being needed on the Windows server - the non-prod SSH key and the prod SSH key.

  2. All the vital SSH 'pem' files needed for connecting to both non-prod and prod servers are stored in one place - in the AWS Parameter Store in an Encrypted format. Much more secure! What's more, IAM Roles are restricting the jump box servers to what files they are allowed to read - i.e. the non-prod jump box can only read non-prod files, the prod jump box can only read prod files.

  3. It makes SSH key management much easier.
    Since we know that there is only one source of truth for all the SSH 'pem' files, it becomes much easier for us to change / rotate SSH keys since they are not stored on local servers. In these cases, all we need to do is generate new SSH 'pem' keys on a target server and then update the old 'pem' file in the AWS Parameter Store with the new key. Easy.

  4. No artifacts are required - or left behind - on the jump boxes.
    Again, much more secure.

The last thing I'll add here is ensure that you have backed up / saved all your SSH pem files to a very secure place - for example, Zoho Vault or 1Password.

I hope that this is useful to you in your work to secure your AWS environments.

Adrian Nakon

Read more posts by this author.

Melbourne, Australia.