Design of a Docker Logging Driver
• Mark Eschbach
Recently the Irrigation Let’s Encrypt instance failed to renew the Let’s Encrypt certificate for one of my domains. To get the logs of the instance I will have to log into the servers and extract the logs. I have yet to find time if Kubernetes on metal would resolve this problem however I thought it would be interesting to write a plugin for Docker.
At least for these types of plug-ins they are container instances. From my current understanding Docker will look at config.json in the current directory to discover what features the plugin provides. The example implementation provided does a great job of showing how to use the Go SDK for integration, however I am left wanting for implementing something via NodeJS.
config.json provides the name of the driver socket to utilize prefixed with
/run/docker/plugins/. For example, if the socket used was
logging-postgres.socket the full path would be
/run/docker/plugins/logging-postgres.socket. This can be seen with the
Logz.io implementation. The protocol is HTTP over Unix Domain
Logging Lifecycle management
Built on top of the HTTP/Unix socket is an RPC style interface. For the logging protocol only two resources are required:
/LogDriver.StartLogging- Instructs the plugin how to consume logs from a new container.
/LogDriver.StopLogging- Instructs the plugin to stop consuming logs from a specific container.
Much to my annoyance the actual verb utilized isn’t specified in the documentation or SDKs. This means we’ll have to use a catch all in the target language.
When requesting to start logging a container the JSON body passed describes the unix socket to consume the log entries and meta-information about the container being logged to. If the logging application is unable to keep up with the incoming stream the container will become blocked on writes to standard out.
The socket produces a length delimited stream. Length is a 4 byte unsigned integer followed by the data cells. Each cell within the stream contains a Protobuf LogEntry message.
Shutting Down Logging
When requested to shutdown logging, the
/LogDriver.StopLogging resources will be invoked with the unix socket path to
finish. Once the driver responds to the stop request Docker will then remove the underlying socket. There is a
warning about consuming all messages on the stream however there is no information on how to determine what is the last
message. Perhaps the
LogEntry.partial_log_metadata.last flag? This will require some experimentation.