This ZenPack is developed and supported by Zenoss Inc. Contact Zenoss to request more information regarding this or any other ZenPacks. Click here to view all available Zenoss Open Source ZenPacks.
This ZenPack provides support for monitoring Docker containers running on Linux devices.
Docker versions 1.6 through 1.11 are known to be supported as of June 20th, 2016.
The features added by this ZenPack can be summarized as follows. They are each detailed further below.
The following components and properties will be automatically discovered when the DockerCollector modeler plugin is enabled for Linux devices running Docker.
This information is obtained by running the following commands on the Linux device via SSH.
As with all SSH modeling, the zCommandUsername, zCommandPassword, and zKeyPath configuration properties are used to establish the SSH connection. For password authentication, zCommandUsername and zCommandPassword must be set. For public key authentication, zCommandUsername and zKeyPath must be set. The value of zCommandPassword will be used as the passphrase if the key file provided by zKeyPath requires a passphrase.
Note: The DockerCollector modeler plugin is not enabled by default for any device classes. Typically you would add it to the /Server/SSH/Linux device class so all Linux devices being monitoring via SSH will have any containers discovered. Alternatively you could create a /Server/SSH/Linux/Docker deviceclass under which you placed all of your Linux devices that operate as Docker hosts, and only add the DockerCollector modeler plugin for it.
Three types of monitoring are performed for all discovered Docker containers: Status, statistics, and size. The monitoring of each of these aspects has been separated so that you can choose which you are interested in monitoring more easily.
Note: The default value of zCommandCommandTimeout is 15 seconds. This may not be long enough for the monitoring commands detailed below to execute. It is recommended that this value be increased to 60 seconds if status and statistics monitoring are enabled. It is recommended that this value be increased to 600 seconds if size monitoring is enabled.
When either zDockerMonitorContainerStatus or zDockerMonitorContainerSize are enabled, the status of the docker daemon will be checked once per minute (for zDockerMonitorContainerStatus) or once every ten minutes (for zDockerMonitorContainerSize) by running one of the following commands respectively.
An error event such as the following will be created if either of these commands results in an error instead of a list of containers.
A clear event such as the following will be created if the above commands properly result in a list of containers.
The ZenPack installs a docker-ps-status event class mapping into the /Status event class to handle these events by default. You can create an alternative mapping for the docker-ps-status eventClassKey with a lower sequence number if you wish th handle these events differently.
When zDockerMonitorContainerStatus is enabled, the status of each container will be checked once per minute by running the following command.
Containers with any status other than "Up" or "Created" will result in a critical event being created for the container with the following example fields.
Containers with an "Up" or "Created" status will result in a clear event being created for the container with the following example fields.
The ZenPack installs a dockerContainerStatus event class mapping into the /Status event class to handle these events by default. You can create an alternative mapping for the dockerContainerStatus eventClassKey with a lower sequence number if you wish to handle these events differently.
Note: Container status monitoring is disable by default because container down events will only auto-clear if the same container is restarted. If the container is left in a non-running state, or if is removed, its event must be manually cleared. If auto-clearing is important you may want to consider using Zenoss' normal process monitoring support to monitor the process(es) running within the container instead of monitoring the container.
When zDockerMonitorContainerStats is enabled, the statistics of each container will be collected once every five minutes by running the following commands.
These three commands capture the cgroup CPU, memory, and IO usage respectively. They will work regardless of whether the cgroupfs or systemd driver are used.
The following datapoints are parsed from the results of these commands.
The following graphs are built using these datapoints.
When zDockerMonitorContainerSize is enabled, the real size and virtual size of each container will be monitored once every ten minutes by running the following command.
Note: zDockerMonitorContainerSize is not enabled by default. The reason for this is that adding -s flag to docker ps can result in the command taking a very long time to run when many containers, or large containers are used. It is recommended that you attempt to run the command on your Docker hosts manually and see that it takes less than 10 minutes to execute before enabled zDockerMonitorContainerSize.
Note:Older versions of Docker only report real size, not virtual size. The exact version cut-off isn't known, but Docker 1.6.2 as known to not report virtual size.
The following datapoints are parsed from the results of this command.
The following graph is build using these datapoints.
When combined with the Zenoss Service Dynamics product, this ZenPack adds built-in service impact capability for Docker Containers. The following service impact relationships are automatically added. These will be included in any services containing one or more of the explicitly mentioned entities.
To begin discovering and monitoring Docker containers you must start by deciding what kind of monitoring you want to perform. There are three configuration properties that will control how container monitoring is performed.
You can refer to the various container monitoring sections above to understand exactly what the implications of each of these properties are. By default only zDockerMonitorContainerStatus and zDockerMonitorContainerStats are enabled. This is due to a potential performance consideration when monitoring container sizes. See the note above in the Container Size Monitoring section.
After setting these configuration properties to the desired values, you must enable the DockerCollector modeler plugin for the device class(es) or device(s) for which you want to discover running containers. One possibility would be to create a /Server/SSH/Linux/Docker device class, and add DockerCollector to the list of modeler plugins it will inherit from the /Server/SSH/Linux device class.
Docker container discovery and monitoring will occur by running commands on the monitored device with SSH. This SSH connectivity will use the same SSH configuration that is used for normal Linux device monitoring. The following configuration properties can be used to control SSH access.
See the Discovery section for more information on how zCommandUsername, zCommandPassword, and zKeyPath are used. See the Monitoring section for a special note on zCommandCommandTimeout.
In addition to SSH access, this ZenPack executes specific commands via sudo both during discovery and monitoring. This means that sudo must be installed on the monitored system, and if zCommandUsername is not root, sudo must be configured to allow the user specified in zCommandUsername permission to run the following commands without specifying a password.
It is also important that sudo be configured to allow running commands without a tty. Specifically this means that a line such as the following in sudoers (visudo) can prevent discovery and monitoring.
You can resolve this by disabling the requiretty option for all users by changing the above line to the following:
You can also selectively disable requiretty just for the user configured in Zenoss' zCommandUsername configuration property. Assuming that username was zenmonitor, this is how that would look.
This ZenPack requires the following services (daemons) to be running.
Installing this ZenPack will add the following items to your Zenoss system.