…often run awry. Within our release pipeline our application’s version is bumped. I’ve tested this code by hand multiple times and seen it work in the postive and negative case. The release process verifies the release tag is at the HEAD of the branch before starting to make changes. We intended this to prevent strange case of changing the branch to the wrong version. The change itself wouldn’t be a big deal but going backwards even on patches it makes rationalizing releases.

Unforunately last night we encoutnered a failure case. Looking at the branch history this morning there were not commits after the tag on that branch. So perhap the check itself is wrong?

  echo "Finding head"
  GIT_HEAD=`git show-ref -s $DEPLOY_BRANCH`
  echo "Found head"
  if [ "x$DEPLOY_COMMIT" = "x$GIT_HEAD" ] ; then
    bump_minor_version
  else
    complain_loudly
  fi

The -s option is intended to resolve the branch name into the actual SH1 commit. The DEPLOY_COMMIT is adaption from Travis CI’s envrionment variables about the run. DEPLOY_BRANCH is reverse engineered from the local verison then verified against the repository. Perhaps the code chose the wrong branch?

After spending some time on it I can’t conclusively say what occurred. I’ve put some additional checks, provided some additional details, and made the script louder. Sometimes that is the best we can do.

Considering Vault and Consul

Onto the baseline work! Time to prototype Vault backed by Consul for HA. I’m hoping to setup the system in a way where it doesn’t make a difference to developers but will take over in production. After my prior reading I could start with Vault then add HA via Consul, but Consul is the highest risk since I have no expierence with it. Most of my expierence has been with Etcdv2 which exposes it’s information over an HTTP interface. Looks like Consul communicates over both HTTP and DNS. Their architecture internals seems to cut through the fluff and gets to the brass tacks of the system.

Interesting at the core they say Consul uses Raft for consensus however it’s implemented in Serf. I’ll need to look into the details. I am glad they make a reasonable distinction between local data center traffic and multi-data center traffic. Like Etcdv2 they seem to recommend a small number of servers to reduce the amount of time to reach consensus. Interestingly the documentation seems to play off all nodes which aren’t the leader as a proxy.

AFter reading through the remainder of the documentation the Consul system is fairly straight forward. The system feels like a reasonable wiring to resolve the distributed discovery problem with a reasonable level of confidence and did a much better job than Java’s RMI Registry. Unforunately my main concern around Consul is the system’s requirement I run client nodes on all machines consuming the services. I’m generally concerned this is impractical due to some services not being under our direct control (RDS) or trying to only run critical software (ECS hosts).

The primary goal of running Consul is to put our Vault instance into an HA configuration. At this point given the complexity of getting Consul bootstrapped is going to be a pain for external clients. Nothing like giving this system a spin to see what happens.