By Amir Naftali.
I was looking for ways to tighten security on my Amazon EC2 servers. EC2 is so easy to use that it made me almost forget that each port I open on my servers is accessible to me — but also everyone else — via the public internet.
My goal was to create a flexible development and integration environment (and possibly production) by allowing myself to connect to my EC2 servers from everywhere with my debugger, database client and test tools, run diagnostics, CI servers and so on, without having to worry that each port I’ll open will require an entire security evaluation and hardening.
Architecture-wise, I thought of creating a VPN Gateway (of course it will be a software VPN Gateway) that will be a single point of entrance to my EC2 public cloud. The VPN Gateway will serve as a termination point for the VPN tunnel and a simple software router for forwarding and receiving traffic from the internal AWS network environment. Access to the Gateway over the public internet will be secured and the Gateway will forward traffic to and from my EC2 servers (protected by Security Groups). This way (by using EC2 Security Groups) there will be no need to open ports on my EC2 servers to the public internet.
Technology-wise, I explored different approaches, such as SSH and OpenVPN, which could be decent solutions, but I was looking for something a bit more generic.
I thought that because IPSec allows Layer 3-based encryption it would be a good idea to use it to build a Remote Access VPN. So using an IPSec gateway in EC2 will allow me to disconnect my EC2 servers form the public Internet and only allow communication to and from the trusted (GW) IP address. As I researched this, I found that a few other folks have attempted this but some had given up and went for other approaches. Nonetheless, I decided to dig in a bit more and here is what I found (including an example of how to do it).
Before I go on, if you don’t want to get your hands dirty and you would like to speed things up, 40Cloud can automate this process for you. For quick and easy installation of the 40Cloud solution, use the 40Cloud AMI or BYOL available on the AWS Marketplace.
Manually setting up IPSec is more challenging than using SSH or OpenVPN, but there are strong advantages to IPSec over other technologies (I’m not going to argue for one over the other here because the discussion depends heavily on the use case and environments).
Getting IPSec to work in a public cloud environment requires a bit more effort than in a controlled environment, but it’s definitely doable (example below).
There are a few things you’ll need to know before setting up an IPSec gateway in Amazon EC2:
- EC2 (but not Amazon VPC) only allows TCP/UDP and ICMP traffic to/from server instances. This means that both IPSec’s ESP and AH headers (as many other protocols) are blocked by the network infrastructure. The only way to get IPSec to work in this environment is to use the encapsulation model, which sends IPSec packets over UDP. This has some nice advantages such as the fact that the cloud provider will not be able to know which protocols you are using between servers and you’ll be able to work around the protocol limitation forced by the environment. This way, you’ll be able to use all protocols and not just TCP,UDP and ICMP. The downside is that IPsec/UDP will reduce your MTU by a few bytes.
- If you want to get road warriors to work with your IPsec gateway you need to be aware that EC2 is fully NATed per region. This will create challenges for Windows machines, because the default IPsec policy on the standard Windows client does not allow connecting to an IPsec server that resides behind NAT. So to get your Windows servers to work, you’ll need to tweak the Windows registry to support this (note that this is a Windows-only challenge, NATed GW will work fine with OSX and Linux clients).
- EC2 Security Groups: To get IPSec to work you’ll still need to open on the Gateway few UDP ports to the public internet (or to a specific location you would like to access from). The UDP ports are 4500 (IPsec/UDP), 500 (IKE) and 1701 (L2TP).
Generally, not just EC2-related, you should also keep in mind that:
- IPsec works well for point-to-point connections. It encrypts/decrypts traffic between the two points with little overhead. For road warriors it’s a bit different, because in many cases packet overlaying is required and IPsec doesn’t provide this functionality, so you need to stack a few things to get it to work. Most “industrial” clients do this for you, but it’s still good to know how things work under the hood.
- L2TP/PPP – in most cases the industrial client will stack L2TP with IPsec. L2TP is a Layer 2 tunneling protocol over UDP (port 1701) and stacked with PPP it provides an overlay IP-to-IP connection. When running with IPsec it provide the overlaying layer (in some cases PPP is used for authentication).
A Step-by-Step Configuration Example
My assumption here is that things like EC2 Security Groups and Elastic IPs are known terms and don’t require further explanations.
Basically, this is what I wanted to accomplish:
In the above, all packets between my laptop to the IPSec GW (traveling over the public Internet) are secured (encrypted and authenticated), and packets flowing between my GW and my EC2 servers (traveling inside the EC2 region) are secured using EC2 Security Groups .
The following screen shot depicts how the complete setup looks like in the 40Cloud web console . The VPN Gateway (the big orange shield icon) connects one VM (a server icon with small orange shield), all located at the AWS data-center in Ireland. The IP subnet allocated for connecting road-warriors is 126.96.36.199/24 (shown as three little orange men figures). The 40Cloud web console allows monitoring of the cloud deployment in real-time as well as logging all system events for future analysis and configuration of all security related items.
In this example I am building the gateway on an Ubuntu system.
Launching a Server
First thing you’ll need to do is to launch an ubuntu server either from the EC2 management console or from command line. (just don’t use 13.04 since it looks like there is a regression bug related to IPSec. I used 12.10 and 12.04 will also work fine). I’m using a micro instance but you can allocate a server from any size that fits your need.
Assigning an Elastic IP to the Gateway (GW)
Eventually I’ll need to allocate an Elastic IP to the server. My suggestion is to allocate it now in advance as an initial step but you can also do it later (but it must be done before connecting to the Gateway).
Allowing SSH to the GW
Make sure that the EC2 Security Groups assigned to the Gateway allows inbound SSH traffic (port 22).
login to the Gateway, using SSH and follow the next set of instructions
Setting up the VPN GW (in a few simple steps)
Generally you will need to setup both the IPSec and L2TP/PPP protocol stacks, turn the GW into a virtual L3 router and then tweak a few things with EC2 to make it work.
For L2TP I’ll be using xl2tpd which is available for most platforms.
For IPSec/IKE I’ll be using the well-known OpenSwan.
The examples for config file I provided should just work (but you’ll need to replace the <server ip address> in the different files with the instance private ip address provided by EC2).
It is highly recommended to change the shared secret and passwords from the one I’m using in my examples (if not before, right after you get things to work)
Step 1- Setting up xl2tp and PPP
First you’ll need to install (if not already installed) the l2tp daemon (i’m using xl2tpd).
To install xl2tpd just type
sudo apt-get install -y xl2tpd
Here is an example for an xl2tpd configuration file that works (you’ll need to fill in the <server ip address> with the instance private ip address.
Download an example of /etc/xl2tpd/xl2tpd.conf (xl2tpd config file):
Download an example of /etc/ppp/options.xl2tpd:
To make sure the xl2tpd stack will use the updated configuration files just run the following commands
- service xl2tpd stop
- service xl2tpd start
xl2tpd is now running with your configuration (you can check /var/log/syslog to verify it was loaded successfully).
Step 2- Setting up user credentials for user authentication
L2TP with PPP allows you to perform user-based authentication. There are several ways to make this work, but I’ll use the simplest (and a non-secure way of managing users with L2TP/PPP).
Each user will be represented as a line in \etc\ppp\chap-secrets
The file looks like this:
<username> <servername> <password> * user1 MyGW password1 * user2 MyGW password2 * .......
Please note that
- Items are separated by a single tab
- <servername> must be identical to the name provided in xl2tpd.conf
- usernames and passwords are in the clear (as I mentioned, this is not that secure)
- the “*” allows that specific user to access from any IP address
Step 3 – Setting up OpenSwan
See the OpenSwan site for more detailed information.
To install OpenSwan just type the following:
sudo apt-get install -y openswan
(If asked by the installation process just answer “NO” to every question).
Configuring OpenSwan connections
Here is an example for an openswan ipsec.conf configuration that works (you’ll need to fill in the <server ip address> with the instance private ip address.
Download an example of config /etc/ipsec.conf:
Configuring OpenSwan authentication
In this example i’ll use shared secret, which is the simplest way possible to authenticate the server, and will require writing only a single line in ipsec.secret.
Download an example of /etc/ipsec.secrets:
To get ipsec to work just type the following commands:
- service ipsec stop
- service ipsec start
- ipsec auto – – add RWConn
A few things to note:
- There are a few places in the different config files above where you’ll need to provide the <server ip address>. This is the EC2 internal IP address provided by Amazon and not the external public IP address (or Elastic IP). this address can be obtained by typing “ip addr” and copying the IP address associated with eth0, or from the EC2 management panel.
- The <name> in the xl2tpd.conf file must be identical to the <name> in options.xl2tpd
- The subnet provided in the ipsec.conf file (in my example virtual_private=%v4:172.24.0.0/13) must contain the “ip range” subnet in the xl2tpd.conf file (the “ip range” subnet must start and end with a valid host ip – no subnets should be used for stating start and end address)
- the “local ip” in xl2tpd.conf must be a valid host IP and must be outside of the “ip range” subnet (but in the subnet defined in the ipsec.conf by virtual_private parameters) otherwise one of the connecting hosts might be assigned with the same address as the server
Step 4 – Configuring EC2 Security Groups
Before we can test the GW we must open a few holes in the Security Group to allow creating IPSec connections.
Note that EC2 Security Groups do not control which protocols are carried within the IPSec and the L2TP/PPP stack so basically you can bypass the TCP/UDP limitation EC2 has using this setup.
We’ll need the following rules for the GW inbound traffic:
UDP port 1701 (for L2TP)
UDP port 500 (for IKE)
UDP port 4500 (for IPSec over UDP)
Step 5 – Turning your Gateway into a Software Router
Now that we’re all set for securing the traffic over the public internet and into the region, all that is left to do is to get the GW to forward packets to/from other instances in the region.
There are two things we need to configure here:
- Packet forwarding
- NAT-ing packets – this is critical since the EC2 network can’t route packets that originate from outside of the EC2 region back to the Gateway, so the GW must masquerade all packets as if they were generated by it.
Configuring packet forwarding
This is very easy (thanks to Marius Ducea @mariusducea), just type…
sysctl -w net.ipv4.ip_forward=1
In order to persist this change, you can also make the following change in /etc/sysctl.conf:
net.ipv4.ip_forward = 1
And then execute the following command:
sysctl -p /etc/sysctl.conf
And you are all set (you can read more about IP forwarding here).
Masquerading will be done by IPTables (Information on IPtables is widely available, I personally like this one). Here is the basic config:
iptables -F iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -A POSTROUTING -j MASQUERADE
Testing and Securing your other EC2 servers
Now you are all set. Just configure your VPN client to use IPSec/L2TP (most operating systems have a native client already installed) and
1) GW address should be the Elastic IP address that you assigned to the GW
2) The shared secret to use is the one that was configured in ipsec.secret file
3) Username and password to use were configured in chap-passwords file
4) You may wish to avoid working in split tunnel mode, so that all traffic should go through the VPN GW. Otherwise you might hit a conflict with your local private IP address subnets.
Once the VPN connection is established try to SSH your other EC2 GW – it should work for you.
Now that we have everything working, we can significantly improve our EC2 server security by:
- Allowing developers/tester/devops to access EC2 only via the GW, which means NO MORE opening of ports to the public internet for access – To do so, just change your EC2 servers security groups to only allow traffic from the GW and remove the rules that allow access from the public internet.
- Control access at user level (denying/allowing access now only requires removal/addition of a user from/to the list of users, so the burden of public key management with EC2 is relaxed)
- Authenticate and encrypt all admin traffic over the public Internet regardless of the protocol being used or location the traffic is coming from.
This is pretty much it. Nice and easy 🙂
We would love to hear your feedback on this post.
If you need further assistance, we’re happy to help.