Dynamically reconfiguring Nginx upstreams using Etcd
My goal is rather simple: having nginx reconfiguring it's upstreams when a service changes it's deployment locations. I've hit several snags along the way. In the current cloud configuration I have an agent which automatically updates static site contents based on etcd keys, including reconfiguring nginx. I hit a limitation with migrating application services pretty quickly. Upon restart, upstreams must be defined otherwise nginx panics.
Prototype, Mark 1
The first prototype design was simple: upon notification a configuration JSON document and application port bindings would be applied to a template to produce the host configuration. The JSON document and application ports would be stored in etcd and upon notification update nginx host configurations. I chose Ruby for the environment due to my primary constraint being on development time. In a fairly short time the program was up and running, dyanmically generating new nginx hosts and upstreams using a command line interface.
The resulting etcd tree would appear as follows:
-
Key Prefix
In testing this was /lb. On production systems a prefix of the cluster name would probably be a better choice (i.e. /aws-west2/lb). Unlikely as it would be to run etcd2 globally, it ensure I don't bork something in a tired state.
Upstream name
- config
- JSON document containing configuration info
- upstream-{id}
- Right hand side of upstream server's statement to be injected into the nginx configuration
So the drawbacks so far are that etcdctl can store JSON docuemnts; the tooling gives the impression I might run into issues later. There are quiet a few path calculations and a recursive scan against etcd. The deplyoment scenario requres etdctl's exec-watch feature to execute the ruby script, which should be an advantage as I'm using etcdctl with on the services on the boxes.