Mark Eschbach

Software Developer && System Analyst

Dynamically reconfiguring Nginx upstreams using Etcd

My goal is rather simple: having nginx reconfiguring it's upstreams when a service changes it's deployment locations. I've hit several snags along the way. In the current cloud configuration I have an agent which automatically updates static site contents based on etcd keys, including reconfiguring nginx. I hit a limitation with migrating application services pretty quickly. Upon restart, upstreams must be defined otherwise nginx panics.

Prototype, Mark 1

The first prototype design was simple: upon notification a configuration JSON document and application port bindings would be applied to a template to produce the host configuration. The JSON document and application ports would be stored in etcd and upon notification update nginx host configurations. I chose Ruby for the environment due to my primary constraint being on development time. In a fairly short time the program was up and running, dyanmically generating new nginx hosts and upstreams using a command line interface.

The resulting etcd tree would appear as follows:

  • Key Prefix

    In testing this was /lb. On production systems a prefix of the cluster name would probably be a better choice (i.e. /aws-west2/lb). Unlikely as it would be to run etcd2 globally, it ensure I don't bork something in a tired state.

    • Upstream name

      JSON document containing configuration info
      Right hand side of upstream server's statement to be injected into the nginx configuration

So the drawbacks so far are that etcdctl can store JSON docuemnts; the tooling gives the impression I might run into issues later. There are quiet a few path calculations and a recursive scan against etcd. The deplyoment scenario requres etdctl's exec-watch feature to execute the ruby script, which should be an advantage as I'm using etcdctl with on the services on the boxes.