Monit How-To: Getting Started With Process Monitoring
Monit is a versatile open source tool that allows you to monitor and control application state. If something’s wrong, Monit is always there to save the day. It can automatically start a process if it’s not running, restart one if it’s not responding and stop a process if it’s consuming too much resources. Further, you can manually configure Monit behavior for emergencies. This enables you to always know if the application has started to fail and quickly come up with the right solution.
In this article, we’ll be guiding you through Monit setup and configuration. You will also learn how to pair it with Sidekiq, Nginx, Puma, and Elasticsearch.
Installing Monit
Since we will be using Ubuntu Server, the installation procedure is pretty simple. Monit is available in the official repositories:
After Monit is installed, you need to modify a few things in the main configuration file located at:
/etc/monit/monitrc
However, you will need to uncomment the following lines first:
#set httpd port 2812 and
# use address localhost # only accept connection from localhost
# allow localhost # allow localhost to connect to the server and
Now you can restart the service:
You can check if Monit has loaded the configuration file and is running correctly:
The output data should look something like this:
➜ ~ sudo monit status
The Monit daemon 5.6 uptime: 0m
System ‘my-virtual-machine’
status Running
monitoring status Monitored
load average [1.02] [1.08] [1.04]
cpu 0.0%us 0.0%sy 0.0%wa
memory usage 2885568 kB [74.0%]
swap usage 3075448 kB [76.0%]
data collected Fri, 15 Jul 2016 17:08:28
Monit’s logs are available at the default location “/var/log/monit.log”. You can change the location address in the configuration file by modifying the following line:
set logfile /var/log/monit.log
Enabling the web interface will provide you with basic GUI and control options — the same ones available in the console.
To do that, you need to slightly modify the configuration file.
First, uncomment the line:
# allow admin:monit # require user ‘admin’ with password ‘monit’
Here, you need to set the username and the password that will be used to access the web interface. It’s strongly recommended to use a complex password since the web interface has full access to Monit and services.
Also, you need to comment back the previously uncommented lines:
# use address localhost # only accept connection from localhost
# allow localhost # allow localhost to connect to the server and
This will allow you to connect localhost — as well as any other external IP — to the Monit daemon, so be careful.
After you finish modifying the configuration file you need to restart the Monit service:
After the web interface has been configured it can be accessed by this link.
The port address can also be changed to a custom one if security is a top priority.
How to configure alerts in Monit
If you want to always have the latest updates on the monitored processes — just enable alerts and Monit will start sending emails to a specified address should any errors occur. To enable this option you need to change few lines in the main configuration file.
set alert email_address
By default, Monit uses local host for sending emails. Further, you can specify several mail servers that will be used to deliver alerts. If the first one fails, Monit will move to the second one and so on. Here is how you can enable this option:
set mailserver some_mail_server
Monit configuration and setup
In commercial projects development, we use Monit to monitor and/or control services like Sidekiq, ElasticSearch, Redis server, Rpush, Nginx, PostgreSQL, and some others. We will show you how to set up each of them starting from nginx.
Nginx
First, you need to create a configuration file in the “/etc/monit/conf.d/nginx” directory.
check process nginx with pidfile /var/run/nginx.pid
start program = “/etc/init.d/nginx start”
stop program = “/etc/init.d/nginx stop”
Basically, what we do here is specify the location of the Nginx process’s PID (check process “name_of_the_process process” with pidfile) to check if it’s available.
The next two lines enable you to control Nginx through Monit and give you the ability to start/stop/restart service.
After you’ve finished the configuration procedure, you need to save the changes and reload Monit to make them work:
To check if everything has been applied correctly you need to use the following command:
Process ‘nginx’
status Running
monitoring status Monitored
pid 1326
parent pid 1
uptime 13d 1h 55m
children 4
memory kilobytes 3956
memory kilobytes total 29792
memory percent 0.0%
memory percent total 0.1%
cpu percent 0.0%
cpu percent total 0.0%
data collected Fri, 15 Jul 2016 22:36:22
Now you can start/stop/restart Nginx through Monit by running the following commands:
But what if you need to control and monitor a Rails application and its services? E.g. you want to get updates on the state of your Puma web server or restart a Sidekiq instance after updating something. We’ll show you how to do that!
Sidekiq
Sidekiq is a background processing framework for Ruby. We also use Monit to control and monitor Sidekiq state. Why? It is very flexible and enables you to do the same thing in several ways through a config file:
check process sidekiq
with pidfile “/home/deploy/apps/staging.com/tmp/pids/sidekiq-0.pid”
start program = “/bin/su — deploy -c ‘cd /home/deploy/apps/staging.com && bundle exec sidekiq -e staging -C /home/deploy/apps/staging.com/config/sidekiq.yml -P /home/deploy/apps/staging.com/tmp/pids/sidekiq-0.pid -L /home/deploy/apps/staging.com/log/sidekiq.log -d’” with timeout 90 seconds
stop program = “/bin/su — deploy ‘cd /home/deploy/apps/staging.com && bundle exec sidekiqctl stop /home/deploy/apps/staging.com/tmp/pids/sidekiq-0.pid’” with timeout 90 seconds
if totalmem is greater than 400 MB for 2 cycles then restart
if does not exist then alert
What we do here is tell Monit that there is a Sidekiq instance running and then specify the PID file location.
check process name_of_the_process with pidfile “location_of_the_pid_file”
If you save the changes and reload Monit now, it will grab the process and tell you if it’s running or not. But what will happen if you start/stop it through Monit?
start program = “command_to_execute_to_start_the_process”
stop program = “command_to_execute_to_stop_the_process”
This applies to Nginx or any other simple system process, but if there are some Rails application components, you need to specify how the process will start in detail. In the code above, we tell Monit that it needs to execute the command with user “deploy” and that will start the process.
Puma
Puma is a web server for Ruby applications. We use Monit to monitor and manage Puma through this config:
check process puma_production
with pidfile “/var/www/production/shared/pids/puma.pid”
start program “/bin/su — deploy -c ‘cd /var/www/production/current && /usr/local/rbenv/bin/rbenv exec bundle exec puma -C /var/www/production/shared/config/puma.rb’”
stop program “/bin/su — deploy -c ‘cd /var/www/production/current && /usr/local/rbenv/bin/rbenv exec bundle exec pumactl -S /var/www/production/shared/pids/puma.state stop’”
group production_ruby
In this example, we are doing pretty much the same as in the Sidekiq part.
You can see that Monit is monitoring the state of the process based on the PID file. We send the start and stop commands as we would normally do this in the console. You might also have noticed group which gives us the ability to control the process by querying the group. Now all the processes controlled by Monit and containing the ”group production_ruby“ line can be restarted by “monit -g production_ruby restart all”. In this case, it’s a useful option since there are multiple processes (Sidekiq, Puma, Rpush) that we need to restart with only one command right after deploy.
Rpush
Mainly, we use Rpush as a notifications service for some web applications. Basically, it’s a gem that allows for sending notifications to mobile devices. As you might have guessed, we use Monit to control Rpush as well:
check process production_rpush with pidfile /var/www/production/shared/pids/rpush.pid
start program = “/bin/su — deploy /bin/bash -c ‘cd /var/www/production/current; bundle exec rpush start -e production -p /var/www/production/shared/pids/rpush.pid’”
stop program = “/bin/su — deploy /bin/bash -c ‘cd /var/www/production/current; bundle exec rpush stop -e production -p /var/www/production/shared/pids/rpush.pid’”
if 5 restarts within 5 cycles then timeout
group production_rpush
All the configuratons are similar to the previous ones, we only used:
if 5 restarts within 5 cycles then timeout
Since Monit runs in cycles, it will monitor the application through specified instructions. In the example above, Rpush state will change to “timed out” after 5 unsuccessful restarts in a row. If some of your applications times out, an event will be triggered and Monit will send a notification to the email you specified in the configuration file.
Elasticsearch
If you are a kind of an Elasticsearch maniac — like we are — Monit is the right tool for you.
This configuration is how we monitor Elasticsearch with Monit:
root@test:~# cat /etc/monit/conf.d/elasticsearch
check host elasticsearch_connection with address 0.0.0.0
if failed url http://0.0.0.0:9200/ with timeout 15 seconds then alert
if does not exist then restart
group elasticsearch
check host elasticsearch_cluster_health with address 0.0.0.0
if failed url http://0.0.0.0:9200/_cluster/health
and content == ‘green’
with timeout 60 seconds
then alert
if does not exist then restart
group elasticsearch
First, we specify the address of a required machine as a localhost (address 0.0.0.0). After it’s done, Monit starts checking if Elasticsearch is running and available in each cycle. In case it’s been unavailable for 15 seconds, Monit will send us an email saying something’s wrong and Elasticsearch is down. Monit will also check cluster health by going to “status” (http://0.0.0.0:9200/_cluster/health). If it’s timed out or the “status” field is no longer green Monit will start an “alert” event. Our DevOps team will be notified through an email. Both Monit processes belong to the same group (group elasticsearch) and that gives us the ability to restart both of them simultaneously by executing the following command:
Conclusion
In development, Monit has proven itself as a very useful tool that offers powerful monitoring capabilities. If an application process behaves abnormally (e.g. resource consumption limit) Monit emails our DevOps team. Thus, we are able to fix issues before they start turning into a serious problem. Further, you can add your clients to the notification list so that they could have the latest updates on the development process.