Part #1: Removing SSH 'pem' files from Jumpboxes in AWS - Introduction

Hi Everyone,

During the work of a major Client Project, one issue that was continually on my mind was how we were using SSH 'pem' files on the non-prod and prod jump boxes.

What follows are the notes on my travels towards tightening up security on our AWS Environment. The mindset is towards making any 'hack' attempt as difficult as possible for an intruder. This isn't negative thinking as such, but rather taking a default position that someone 'has' gotten onto the system, so how can I make them waste as much time as possible whilst they are on it.

And of course, the more time they are on, the more opportunities there are for them to be discovered, etc.

Finally, I have broken up this discussion into two parts.
This is because it covers a lot of ground and some of the content is a bit technical (Part #2 will be posted up a little later).

Setting the Scene

Before we dive into SSH keys etc., it's worth taking a quick look at how these servers (jumpboxes, work servers) relate to each other.

How the servers relate to one another

The idea behind the above design is to provide as much 'segregation' as possible between the different environments.
Using SSH private keys (pem files) we can control each 'jump' required to log onto a target server.

A 'pem' file is a SSH private key, so it is a vital resource that should be heavily protected.
Having these sitting in the /home/ec2-user directory did not fill me with joy - especially since security wise this is a gaping hole.

So, what can we do about it?

Being AWS, there is an EC2 service known as the 'Parameter Store'.
This area can be used for all sorts of things like passwords, SSH keys and other parameters.

So the perfect place to be is:

  1. No SSH pem files on jump boxes
  2. No support scripts etc. on jump boxes
  3. No 'articles' left behind on jump boxes when a tunneling operation is performed
  4. Overall, across the AWS server estate, a vast reduction in the number of SSH pem files stored physically on servers. This includes not just servers like 'non-prod- or 'prod' jump boxes, but also the Windows servers where MobaXterm (or it's equivalent) runs.

Please note that although I centre on MobaXterm for this discussion, the same situation exists when using other tools - such as Putty.

The core issue is that these tools require the whole set of pem files for accessing an environment need to be present on the server where MobaXterm (or Putty) is located. If there are multiple of these servers (i.e. 2+ jump boxes - as is often the case), then multiple copies of these keys will be present. And on Windows, this is exacerbated even further if these pem files are stored not in a 'public' place on the Windows box, but in private user directories - e.g. c:/users/<username>/pems.

Very quickly, the control of critical SSH keys becomes a thorny issue, especially for security conscious Clients.
To find a good solution, we'll need to work with both our tooling (MobaXterm) as well as the environments on AWS.

Looking at MobaXterm, although it allows us to execute a command when we log onto a jump box, it does not allow us any flexibility when first connecting to that jump box.

Let's take a look at that now:

MobaXterm

Notice how for (1), we can only specify a SSH Private key? We don't have the ability to execute a script or command that would 'get' that key for us - unfortunately.
I have written to MobaXterm and have asked them to integrate AWS parameter store access into the Application.
Hopefully they will do something like this into the future.

But given what we have ....
The best we can do at the moment is:

  1. * LIMIT * the number of SSH keys we can store on our Windows server (where MobaXterm runs)

  2. Somehow not have any critical information stored on the intermediate jump boxes themselves.

As it happens, we can use the AWS Parameter store to achieve (2), and that is what I'm going to discuss next.

Using up the AWS Parameter Store.

What we want to do is use the AWS parameter store as our secure storage area for the SSH keys and (believe it or not!) an execution script that actually creates the SSH connection.

So, what is the AWS Parameter Store.

If you look in the AWS console in EC2 and scroll down the left hand side services, it appears near the bottom of the list:

AWS Parameter Store Screen

If we Click on 'Parameter Store', this is what we'll see:
(I've created a few example entries since I did not want to show anything real!)

Creating a new entry is very easy - but I found a real gotcha when trying to add SSH pem files - in that when you try and retrieve them, the carriage-return at the end of each line in a SSH private key is returned as a space ' ' !

Of course, this breaks the SSH private key - but there is a way around this - by inserting values via the 'awscli' command on the Linux command line - but more on that later.

So, we can store parameters etc.
But how can we allow access to the values in the AWS Parameter Store?

Read on ...

Allowing access to the AWS Parameter Store

What we need to do is allow our jumpboxes - non-prod and prod, to access the AWS Parameter Store values - but only for the values relevant to that server. For example, we only want non-prod to have access to non-prod keys, and prod to have access to prod keys.

The first thing we need to do is create an AWS Service Endpoint. An Endpoint is, essentially, a way for users of an AWS provided service to access that service. In this scenario, we need to have access to the AWS System Manager.

To set this up, you need to go to the VPC Service and then select 'Endpoints'.

AWS VPC Dashboard

As an example, lets create an endpoint here for the Systems Manager. By clicking on 'Endpoints', you'll be presented with a page like this (sorry for the size - I wanted to fit the whole page since it describes the situation better):

AWS Endpoint

You can see that in this case I have chosen the AWS SSM Service. Also note that you need to place it in the correct VPC and subnets, as well as applying the correct security groups.

After creating it, you'll end up with an Endpoint that looks like this :

AWS Create Endpoint

So now we have an Endpoint, how are we going to allow an Instance like a non-prod jump box to access it?

It's time to do some IAM magic!

What we need to do is create some specific IAM Roles and then associate them with Policies that will allow the access.
So, what I did was create four (4) new Roles:

There are two Roles for each jump box - one for 'read-only' and the other for 'Manager'.

Read-only is pretty self explanatory - an Instance can only READ the values for, say, non-prod, if it has associated with it the 'RxR-NonProd-Read-Parameter-Store' Role.
The 'Manager' Role allows 'put' operations - which is important since we'll need this to upload the SSH 'pem' files.
In fact, it will be the only way we will be able to do this - since trying to do this via the Console doesn't work (as described earlier).

For each Role, there is a user defined Policy - let's take a look at the Policy for the : 'RxR-NonProd-Read-Parameter-Store' Role.

AWS IAM 1

The 'RxR-NonProd-Parameter-Keystore-Manager' Role is slightly different, since it needs to give a more powerful access:

AWS IAM 2

With that done, the last thing we need to do to allow the jumpboxes access to the Parameter Store is to grant the Instances (ie. AWS Servers) access to the appropriate Role.

For example, looking at the non-prod jumpbox, we need to grant it the IAM Role : 'RxR-NonProd-Read-Parameter-Store' so it can access the AWS Parameter store at, say, /example, via the System Manager Endpoint we defined earlier:

AWS Attach Role to Server

And now attach the Role:

AWS Attach/Replace Role

And taking a look at the Instance, we can now see that the Role is attached :

And that's where I'll leave Part #1.

In Part #2 of this series, I'm going to show how you can do the following :

  • Upload SSH 'pem' files into the Parameter Store via the command line.
  • Read those same files back again.
  • Explain how using this approach, we can download and run scripts from the AWS Parameter Store that in turn download SSH 'pem' files from the AWS Parameter Store - leaving no trace whatsoever that they had ever run or downloaded anything!

Adrian Nakon

Read more posts by this author.

Melbourne, Australia.