We’ve began using terragrunt as our way to share Terraform state files. First step to catching up: figure out how to install. At some point I’ll spend the time to reflect on why I would rather not use brew however now will not be that time :-). Grabbing the binary is as simple as going to the release page and extracting the file. Funny, they have binaries of darwin-i386. Pretty easy to install though, just throw it on the path. This is something I feel like the Go community gets correct with their applications: generally self contained.

First problem I’ve encountered is a leaked reference to the AWS credentials another developer is using. I wonder if Terragrunt allows for similar el-expressions as Terraform. Ah, should be as simple as replacing the expression with an environment lookup. I’m going to be honest here, the nested quotes kind of drive me crazy. Fairly easy to overide using: "${get_env("TF_VAR_aws_profile", "default")}".

On to the next issue! I don’t have access to the AWS S3 bucket storing the state. To be honest I’ve never bothered looking in depth to the AWS security policies and buckets. Amusingly the UI claims the modification to bucket permissions were successful…but failed and it doesn’t bother telling you. This has been the case for at least a year and a half now!

Next up, once I verified I could access the bucket: Terragrunt is still trying to use the other problem for some reason. It’s a race to see who locates the underlying issue first! And Doni won. Turns out to have been some locking issue causing the problem.

Back to ECS

Back to working on getting the ECS cluster up and running. I really wish Terraform prevented you from applying the same plan twice to the same account. Then you try to import things and it’s all broken afterwards :-/.

Read the AWS Cloudformation script the problem seems to be the UserData section I was building off of was incomplete. I only set the configuration for the instance however I didn’t start the service (search for UserData). The stack portion I believe is relatively easy. For the region though, this seems much harder than it should be. After reading through a few examples it appeasrs like I may have misconfigured the security. I need to figure out logging to verify.

So after adding my SSH key I determined the underlying problem is the instance not having the AWS credentials needed. I’ve added AmazonEC2FullAccess in an attempt to verify this is the underlying problem. The user on ECS optimized linux instances is ec2-user by the way. Well, that wasn’t the problem. It looks like I can’t get the security token. Ornery voice “WELL there is your problem”: missing the proper permissions. One of these days I’ll learn the permissions system. I need the following on the host instance profile:

  "Version": "2008-10-17",
  "Statement": [
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      "Action": "sts:AssumeRole"

And that is all folks!