Running Tornado in production

Supervisor is a simple yet powerful client/server system to monitor and control processes on UNIX systems.

I have a number of Tornado servers that are already managed by supervisor. I have found it stable and flexible enough for most tasks.

Installing supervisor

The installation is pretty straight forward.

sudo easy_install supervisor

This will install supervisor and all the required dependencies.

Configuring supervisor

Supervisor ships with a configuration file generator. This is a good way to understand all the configuration options that supervisor supports.

Run this command if your interested in generating the default configuration.

echo_supervisord_conf > supervisord.conf

I am going to skip the default configuration file and just write one from scratch.

This is what my /etc/supervisord.conf looks like

[unix_http_server]
file=/tmp/supervisor.sock
 
[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=50MB
logfile_backups=10
loglevel=warn
pidfile=/var/log/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200
user=nobody
childlogdir=/var/log/
 
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
 
[group:myapp]
programs=myapp_server-3001,myapp_server-3002,myapp_server-3003
 
[program:myapp_server-3001]
command=/srv/myapp/myapp_server.py --port=3001
directory=/srv/myapp/
autorestart=true
redirect_stderr=true
stdout_logfile=/srv/myapp/logs/myapp_server-3001.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
 
[program:myapp_server-3002]
command=/srv/myapp/myapp_server.py --port=3002
directory=/srv/myapp/
autorestart=true
redirect_stderr=true
stdout_logfile=/srv/myapp/logs/myapp_server-3002.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
 
[program:myapp_server-3003]
command=/srv/myapp/myapp_server.py --port=3003
directory=/srv/myapp/
autorestart=true
redirect_stderr=true
stdout_logfile=/srv/myapp/logs/myapp_server-3003.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn

As you see I start 3 instances of my application on port 3000, 3001, and 3002. It is highly recommended to front your tornado servers with nginx.

Running Supervisor

The two main scripts that you need to use are supervisord and supervisorctl.

supervisord is the backend script that manages the processes and logs. It is also responsible for ensuring that the the processes are up and running.

supervisorctl is the control script that you can use to talk to multiple supervisord daemons. You can use this script to start multiple services or groups.

Some important commands

Basic commands

• start Starts processes
• stop Stops processes
• restart Restarts processes
• status Status of a process

The not so basic commands

• update Reloads the configuration file
• reload Restarts the supervisord daemon
• add Adds processes/groups
• remove Removes processes/groups
• tail Start tailing a log file