Cloudformation Best-ish Practices: Part 1

Some guideline I have discovered after a year of working with cloudformation

Elliott Gorrell

5 minute read

Cloudformation is a great way to manage the creation and updating of your resources in AWS, however there are definitely right and wrong ways to do things and gotchas that can get you into weird states if you aren’t aware of them. This guide is titled Best-ish as these are simply practices I have found to prove work well as a rookie cloud engineer and I think there are definitely practices which can be improved (#3 especially!) so feel free to leave a comment with any suggestions you have from your work with cloudformation.

1) Nested Stacks are awesome

This is my preferred approach for structuring my projects - I find it keeps my cloudformation files nicely concise and readable while maintaining a nice structure when you have larger stacks which might contain multiple applications.

It looks something like this:

/infrastructure
  /params
    dev.yml
    prod.yml
  /cloudformation
    master.yml
    /security-groups
      groups.yml
      rules-app-1.yml
      rules-app-2.yml
    /app-1
      master.yml
      application.yml
      database.yml
      load-balancer.yml
    /app-2
      master.yml
    /app-3
      master.yml
      application.yml

The root level master.yml imports all the nested master files. This allows anyone reading the repo to easily see what applications are deployed as part of this stack and also easily look at their relevant resources as everything is nice and modularised.

2) Security comes first

Although security is always important, we are talking about a different kind of security here - Security Groups! As you can see in the dir structure above I like to create all my security groups in one stack and then define all my rules in one more other stacks. I started off the way most people do by defining security groups with inline rules inside the cloudformation file containing the relevant servers. However you start to realise very quickly this gets you into dependency hell as the order in which resources are created becomes chaos.

For example say you have two stacks:
     1) The first with with an Auto Scaling Group of servers and a RDS Instance
     2) The second with a singular EC2 Instance
Now you have these security group rules EC2 Instance -> (TCP 8001) -> Auto Scaling Group -> (TCP 50000) -> RDS Instance

AWS will parse the cloudformation and realise it needs to create the resources in the following order:

Step Resources Created
1 RDS SG
2 Auto Scaling Group SG
RDS Instance
3 Auto Scaling Group
EC2 Instance SG
4 EC2 Instance

You can see how messy this would get as more resources are added and you would quickly hit a circular dependency error for example if the ASG instances also needed to talk to the EC2 instance as both resources need to exist to reference each other however neither yet exists.

This shouldn’t normally occur as for the sake of the argument this Cloud Engineer has created Security Groups with inline rules which isn’t recommended. If he was to instead create Security Groups and then use the AWS:EC2::SecurityGroupIngress resource to create the rules the order would be:

Step Resources Created
1 Create All Security Groups
2 All Security Groups Rules
All servers and database

This creation order would be the same if the SGs and rules were factored out into different stacks so I do that as I find it makes for more readable code and greater visibility on stack updates.

3) Referencing resources in other stacks

Once you start splitting out your cloudformation into multiple files you will need to be referencing cloudformation resources across stacks. Infrastructure-as-code is still quite young and seems to be going through a similar progression and learning to that of the programming languages we all know and love. We are currently in the “Global Variables solve our problems” phase, however these are even worse as they are account wide global variables!

This will change as the space matures however in the meantime the way cross stack referencing is done is through knowing the name of exports (global variables) from other stacks so a good naming convention is a must.. These are created via the Outputs section in a cloudformation script for creating them and the intrinsic function Fn::ImportValue for retrieving them.

To export a reference I do something such as:

Outputs:
  AppServer1:
  Description:
  Value: !Ref ApplicationServer1
    Export:
      Name: !Sub "${AWS::StackName}-appServer1"

For any values which need to be used by other teams or stacks I keep throwing them up the stacks to the root stack and export it there.

For throwing references around between nested stacks I throw around stack names and using naming convention do imports. To do this I always output the name of a stack in its outputs. Then from that sub group of stacks master I can reference and parse the stack name to any nested stacks which has a dependency on it:

Parameters:
  DependentStackName: !GetAtt NestedStack.Outputs.StackName

Then that stack can use 'Fn::ImportValue': !Sub '${DependentStackName}-appServer1'

This has two major and I mean major! side effects to be careful of:
    1) You are now working with exports so if anything is referencing that export it can’t be deleted
    2) We are evaluating variables containing the values we need for referencing at runtime so aws cloudformation validate-template won’t pick up any typos and you won’t find out until the stack update fails

comments powered by Disqus