Tamming Terraform
• Mark Eschbach
I’ve had a problem for a while with Terraform. At some point it just forgot about a few ECS services. This is particularly annoying when trying to modify related attributes: roles, alarms, etc. I have to go in an manually tweak the configurations in the console, something I hope to avoid. Unfortunately Terraform doesn’t support using import
with aws_ecs_service
s. Really sad. From my perspective it should be rather straight forward, retreiving some basic data with the object identifier being a string which may be created via the name. I grabbed the terraform.tfstate
file and opened it in VIM. I copied a similar service from another lifecycle phase of the application and began splicing in the module differences. Let’s see if this sticks.
At first look this appears to have worked fine. For some reason Terraform wanted to update the ElasticSearch configuration although I’m not sure what the difference is. Hopefully it’s not like the aws_security_group
rule which keeps getting recreated. The Yellow
status in ElasticSearch means the replication rules couldn’t be achieved. I wonder if this is just the AWS implementation or if this is all ElasticSearch products. Well, the application mostly worked. It looks like I may have found some edge cases with with Terraform though. Much better now though since I only have four resources claiming to need to be created.
Syslog diconnections from PaperTrail
I get reports from my systems they disconnect from PaperTrail after a period of time. This is rather unforunate as the only place this message shows up is iin the CloudWatch logs of the system. I found an interesting post on StackOverflow regarding this issue. When I was reading through the Python source the component appears to reconncet. Nope, I’m not sure why I thought that. Not even under Python 3. I think that means I need to wrap it.
The tehcnique is fairly straight forward:
class ReconnectingSysLogHandler(logging.handlers.SysLogHandler):
"""Syslog handler that reconnects if the socket closes
If we're writing to syslog with TCP and syslog restarts, the old TCP socket
will no longer be writeable and we'll get a socket.error of type 32. When
than happens, use the default error handling, but also try to reconnect to
the same host/port used before. Also make 1 attempt to re-send the
message.
"""
def __init__(self, *args, **kwargs):
super(ReconnectingSysLogHandler, self).__init__(*args, **kwargs)
self._is_retry = False
def _reconnect(self):
"""Make a new socket that is the same as the old one"""
# close the existing socket before getting a new one to the same host/port
if self.socket:
self.socket.close()
# cut/pasted from logging.handlers.SysLogHandler
if self.unixsocket:
self._connect_unixsocket(self.address)
else:
self.socket = socket.socket(socket.AF_INET, self.socktype)
if self.socktype == socket.SOCK_STREAM:
self.socket.connect(self.address)
def handleError(self, record):
# use the default error handling (writes an error message to stderr)
super(ReconnectingSysLogHandler, self).handleError(record)
# If we get an error within a retry, just return. We don't want an
# infinite, recursive loop telling us something is broken.
# This leaves the socket broken.
if self._is_retry:
return
# Set the retry flag and begin deciding if this is a closed socket, and
# trying to reconnect.
self._is_retry = True
try:
__, exception, __ = sys.exc_info()
# If the error is a broken pipe exception (32), get a new socket.
if isinstance(exception, socket.error) and exception.errno == 32:
try:
self._reconnect()
except:
# If reconnecting fails, give up.
pass
else:
# Make an effort to rescue the recod.
self.emit(record)
finally:
self._is_retry = False
Let’s see how this works. Reasonable, now it reconnects. Tomorrow I’ll need to figure out how to strip the control characters out of the output.