ASG ECS Scaling: Curtous shutdowns
• Mark Eschbach
I believe where I left off was confirm AWS was able to trigger a Lambda function from a message in a queue. So that leaves me with the following design:
- ASG pushes scale in message onto queue
- CloudWatch triggers Lambda to drain instances
- Lambda pushes delayed message onto queue to termiante instance after timeout
- CloudWatch triggers Lambda to terminate instance on message visibility
- Profit
I’ve got part of the code built for the Lambda. It’s still an unknown what the archive will look like but from the documentation I’ve come across I need to produce it under linux to get the right shared libraries. Easy enough.
Finishing the hooks
Right, I need to know the environment I’m going to be running under. I think Node 6 is the highest version supported which is a bit of a bummer. Let’s find out. As of today (2017-08-17) Node 4.3.2 and 6.10.3 are supported. I’ll be targeting Node 6.10.3 then. I know it supports the let
keyword since I used them on the tcp modifications to the syslogd npm package. Let’s find out if it supports async function and generators. According to RisingStack async
is supproted in 8+: bummer.
Shiny Generators
I got distracted by the discussion of StrongLoop’s view of generators. I realize generators are actually syntactic sugar around iterators which make life easier to implement complex coroutines but they seem like they might make a lot of situations easier. From the outset these seem like a great way to implement state machines which might have a few alternate states. I would like to explore more complicated state machines which involve complex loops.
I don’t think I have any particular state machines right now so I think I’ll punt on finding the version for NodeJS since I’m on BART and T-Mobile doens’t have great network connections in the East Bay tunnels.
Decomposing and cleaning the deregistration
At some point I’ve got to choose a logging setup for NodeJS applications. I’m not sure if the organizational default of writing everything to PaperTrail is appropriate in this case. It will help the general group understand scaling events but there is a large disconnect between instance being terminated and instances of the application running. The application should be telling the listener what it’s intending on doing (IE: draiing EC2 instance {id} with ECS instance {id}
) and the console probably isn’t the best place. But it’s what I got right now…so that is where it goes.
My largest problem with promises in NodeJS is they don’t really resolve the pyramid of doom unless I want a lot of functions. In many cases I only have a single usage for each function and they never seem abstract enough. This pain is made ever larger because most libraries use the node standard callbacks. I’m excited to see how async
functions resolve this issue. Decomposition and refactoring become really painful when using raw promises because there are some many functions floating around.
Taking it out for a spin
I keep forgetting export
doesn’t work in NodeJS 7. I really enjoy it as an easy way to export and exceptions have better method names without having to double up on method names. Well, that is all fixed up in it’s redundant style.
Running the script has now produced Error: connect EHOSTUNREACH 169.254.169.254:80 - Local (192.168.1.154:54184)
with the AWS_PROFILE
set. There was a flag I needed to set. Right, I tweeted about this. In issue 1483 of the aws-sdk
they mention needing to set AWS_SDK_LOAD_CONFIG
in order to have Boto like behavior for credential substitution. Well that fixed it.
Onto the next boulder in the road: I’m not getting expected values back for the ECS-EC2ID relationship. I think this might be the correct time to drop in the debug
package. Just setting DEBUG=*
doesn’t result in any output. I know most of the AWS SDK is autogenerated. I should probably learn how to use it now anyway. Turns out it’s a simple debug("some value %s", "some string")
is how it works.
Hmm, the end of the day and the dregistration works. I’ll have to return on Monday to finish it out, hopefully nothing preempts the work.