[supervisord] Multiple processes in a single container
Introduction
Many times at customer site I have been asked for creating a container full of services, daemons and stuff like a container could be used like a full virtual machine.
After explaining the containers' real purpose, their correct usage and the differences between a container and a vm, the process of containerization could be a real pain and in some cases we could be not able to ensure the fundamental rule of 1 container : 1 process.
In the following example I'll show you the steps that drove me to create a multiprocess container: the postfix container.
The postfix container
Postfix is a free and open-source mail transfer agent (MTA) that routes and delivers electronic mail, intended as an alternative to Sendmail MTA.
The most challenging part of the process was to correctly place the logs for sent emails (by postfix) directly on STDOUT.
Logging to STDOUT ensures that some logs grabber (Fluentd in OpenShift v3) could take these logs and send them to the default logs management system (EFK - Elasticsearch/Fluentd/Kibana on OSEv3).
Unfortunately postfix by default wants to send logs to some rsyslog, that usually saves them to a log file, so I've been forced to create postfix container using multiple processes, violating what should be the best practice for creating a container: 1 process per container.
The three processes involved in the container creation were:
- Postfix
- Rsyslog
- tail -f (on /var/log/maillog)
The last process to run and to leave in foreground, it was (for sure) the tail one. tail process will keep running in foreground while postfix and rsyslog do their job, but, what happen if postfix or rsyslog exits unexpectedly?
Nothing!
Docker container will keep running as soon as the tail process does. So your container orchestrator will never notice the absence of postfix process (unless you have some custom health check).
The solution I found was to use an internal processes orchestrator: supervisord (available in EPEL repo).
Supervisord will be responsible to watch and keep them running your configured processes. If a process exits unexpectedly it will run the process again.
So your Docker's CMD/ENTRYPOINT will be the supervisord command.
But now let's take a look to the Dockerfile itself:
FROM rhel7
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
RUN yum install -y postfix supervisor rsyslog
RUN sed -i "s/inet_interfaces = localhost/inet_interfaces = all/g" /etc/postfix/main.cf
RUN echo mynetworks = 172.0.0.0/8, 10.0.0.0/8, 127.0.0.0/8, 192.168.0.0/16 >> /etc/postfix/main.cf
COPY ./resources/supervisord.conf /etc/supervisor/supervisord.conf
COPY ./resources/rsyslog.conf /etc/rsyslog.conf
COPY ./resources/listen.conf /etc/rsyslog.d/listen.conf
CMD /usr/bin/echo myhostname=$myhostname >> /etc/postfix/main.cf; /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
Looking at 'resources' folder [1], you'll find the config file for rsyslog and supervisord, all the conf for postfix are done in place directly on Dockerfile.
Configuration for supervisord daemon is really simple:
[supervisord]
nodaemon=true
[program:postfix]
command=/usr/libexec/postfix/master -c /etc/postfix -d
[program:rsyslog]
command=/usr/sbin/rsyslogd -n
[program:readlog]
command=/usr/bin/tail -f /var/log/maillog
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
As you can see in the readlog program definition I'm setting the proper stdout for tail to supervisord stdout.
The other two file "listen.conf" and "rsyslog.conf" are related to rsyslog itself and they contains edits suggested by a project-atomic blog post:
Running Syslog Within a Docker Container —Project Atomic
Just for quoting it:
The problem was that in RHEL7 and Fedora we now use journald, which listens on /dev/log for incoming messages. In RHEL7 and Fedora, rsyslog actually reads messages from the journal via its API by default.
But not all docker containers run systemd and journald. (Most don’t). In order to get the rsyslogd to work the way the user wanted, he would have to modify the configuration file, /etc/rsyslogd.conf:
- In /etc/rsyslog.conf remove $ModLoad imjournal.
- Set $OmitLocalLogging to off.
- Make sure $ModLoad imuxsock is present.
- Also comment out: $IMJournalStateFile imjournal.state.
That's all with these simple steps and the help of supervisord you should solve your problems!
I've successfully tested the container in various OpenShift environments, mainly as SMTP server (or with proper customization to the config file, as SMTP Relay server).
For more information please refer to:
[2] https://hub.docker.com/r/alezzandro/postfix/
[3] Postfix container used in a persistent Gitlab OpenShift Template: GitHub - sdellang/gitlab-openshift-persistent
If you have any comments/questions please comment!