CarrollOps

A Blog by Jeremy Carroll

S3 Yum Repos With IAM Authorization

I wanted to get away from some of the Big Data posts I have been doing, and focus on the cloud. When I first investigated moving infrastructure to the cloud, I had a need for software repositories that my instances could download packages from. I currently use CentOS 6.x as the OS for AMI images on EC2. I created the smallest instance I could just to configure an apache server to deliver yum repositories. I then had to manage the security groups, availability, and scalability of the web service. It works, but I wished there were hands-off alternatives.

About 7 months ago I came upon a yum plugin https://github.com/jbraeuer/yum-s3-plugin that would allow me to use an S3 bucket as a yum repository. It’s a fantastic idea as S3 spans all US regions, I don’t have to pay for a 24/7 instance running, and it’s scalable. I’ve used this plugin to great success, but there is one minor inconvenience. For the plugin to work you must configure an access_key + secret_key to the instances you create. I took the approach which I felt was the best compromise between security and ease of use. I created user accounts with Identify and Access Management (IAM) which had very restrictive rights (Only to getObject from my yum s3 buckets). I have been recently tasked with setting up additional cloud infrastructure, and went back to GitHub to see if any progress has been made recently with S3 yum plugins. I then discovered a project which does not require access keys to be installed to the instances https://github.com/seporaitis/yum-s3-iam. Huzzah!

In June, Amazon released a feature where you can assign IAM roles to an instance at creation time http://aws.typepad.com/aws/2012/06/iam-roles-for-ec2-instances-simplified-secure-access-to-aws-service-apis-from-ec2.html. These roles with make available additional metadata in the instance which generates a temporary access_key, secret_key, and token which changes every few minutes to use for API requests. In this case it allows us to make requests to S3 without installing any keys or performing key management.

I have submitted a Pull Request to the original author to address a bug fix, and add two enhancements to the code https://github.com/seporaitis/yum-s3-iam/pull/4. I will be assuming at the time of this writing that you are using the version with my PR attached, or it has been incorporated into the main branch.

Installation

  • Create IAM Role (e.g. ‘s3access’) and set a policy that gives s3:GetObject permissions to that role.
  • Launch instances with this role assigned.
  • Inside the instance:
    • Copy s3iam.py to /usr/lib/yum-plugins/
    • Copy s3iam.conf to /etc/yum/pluginconf.d/
    • Configure your your repository as in example s3iam.repo file.

IAM Role

Inside of the IAM console, choose ‘create role’. Give the name of the role (such as s3yum_access). When it comes time to create a policy, select ‘custom policy’. Use the policy I have defined below. It allows the account the ability to list all S3 buckets and only get from the base bucket, and all sub S3 buckets for our custom yum repository. This may not be the most restrictive policy, but it has worked for me. Please replace bucket-name with the name of your S3 bucket.

{
  "Statement": [
    {
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::*"
      ]
    },
    {
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::bucket-name"
      ]
    },
    {
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::bucket-name/*"
      ]
    }
  ]
}

Creating a Yum repository

I currently use the S3 tools http://s3tools.org/s3cmd packages to rsync local files to S3. Create a local repository structure like your normally would, and then sync it to your S3 bucket. If you do not know how to create a yum repository, there is a lot of documentation on google to help you out. The jist of it is to copy your RPM files into a directory, and then run createrepo on it. After issuing the createrepo command it will create the filelists / repomd.xml for it to be a working yum repo. Then sync these directories to S3 however you wish.

Sample Directory layout

noarch
  noarch/cloud-init-0.6.3-0.5.bzr532.el6.noarch.rpm
  noarch/s3-yum-0.1.0-1.noarch.rpm
  noarch/python-boto-2.5.2-1.noarch.rpm
x86_64
  x86_64/python-PyYAML-3.10-1.x86_64.rpm

S3 tools sync

createrepo noarch
createrepo x86_64
s3cmd sync . s3://bucket-name/

Configuring Yum repository

In my pull request I added a feature from the s3-yum python script I was used to. Add s3_enabled=true to enable a repo to be an S3 repo instead of a regular one. This allows you to mix and match repositories from the web with private local S3 repositories.

[s3-noarch]
name=S3-noarch
baseurl=http://bucket-name.s3.amazonaws.com/noarch
enabled=1
s3_enabled=1
gpgcheck=0

Putting it all together

We now have a configured repository inside of S3 with our RPM packages. We have setup a role to use in launching new EC2 instances. We have granted this role rights via IAM to getObjects inside of our Yum S3 bucket repository. Finally we have installed the yum plugin on the AMI instance which will look up the temporary access keys to use to fetch data from our S3 bucket like a private yum repository. It’s pretty easy to test if this is working or not. Run something like this to verify.

yum clean all
yum search <package which only exists in my custom repo>
yum install -y <package which only exists in my custom repo>

Success will rain down upon you as you have created a scalable package serving infrastructure for all of your AMI’s in any availability zone. Comments / suggestions are welcome.

Comments