June 08, 2011 ( ruby, linux )

Managing and monitoring your Ruby application with Foreman and Upstart

In my last post I briefly mentioned a gem called Foreman. Foreman is a gem that allows you to easily organize and horizontally scale your various processes in a centralized manner. For example, in a typical Ruby on Rails application you might have a few different processes: your application server (Thin, Unicorn, etc), a worker (Resque, Delayed Job, etc), a pubsub server (Faye, Juggernaut, etc). Whatever you’re using, you should be monitoring it to ensure that whatever you spin up, stays up.

There are various tools available to do this sort of thing. In Ruby you have God and Bluepill, in Python you have Supervisord, in C you have Monit, and there are plenty more. The one I want to talk about today however is called Upstart, because Foreman makes it easy to “export to” (setup) Upstart in your production environment. It even makes your development environment a lot more pleasant to work with if you have more than one process to run, which is likely the case.

The reason I started using Foreman is because I often have applications that consist of 3 or more processes. This means that I have to keep 3 or more shell tabs open at all times, and also have to switch between them to see each of their output. With Foreman, this is no longer the case.

Getting started in development

Foreman is very simple to set up in development. All you have to do is create a file called Procfile in the root of your application and define your processes inside of it. Here’s an example Ruby on Rails application using Thin as its app server, Resque as its worker process and Faye as its pubsub server.

web:     bundle exec thin start -p $PORT
worker:  bundle exec rake resque:work
pubsub:  bundle exec thin start -p $PORT -R ./faye.ru

With this in place we are now able to start using Foreman. As you can see we’re executing every command from within the bundled environment using Bundler. This is always a good idea as it isolated each environment and avoids gem version conflicts.

Installing Foreman is as easy as running gem install foreman. With Foreman installed, we can now start all our processes with a single command from our application’s root.

foreman start

You’ll see that Foreman starts up all the processes defined in the Procfile, and also give each of the processes a different color in the console for readability. It looks something like this:

Foreman Example

One of the key features of Foreman is the ability to add concurrency to each of the processes by initializing Foreman with additional parameters. In development this might not be too interesting, but in production it sure is.

foreman start -c web=4 worker=2 pubsub=1

This would spin up 4 Thin instances, 2 Resque instances and 1 Faye instance. By default Foreman will use port 5000 as a starting point. Then it increments by 1 for the corresponding process, and by 100 for each separate process. In the above code snippet that would mean:

  • Thin runs on ports 5000 5001 5002 5003
  • Resque runs on ports 5100 5101
  • Faye runs on port 5200

You can change the starting port using the -p option. It must be a 1000-value (1000, 2000, 3000, etc). Refer to Foreman’s the man page for more information.

From now on when you want to start up your development environment, all you have to do is to cd in/to/your/app and run foreman start. This is really nice, especially if each process has a couple of arguments you would normally need to remember and type in every time to start up your whole environment.

Converting Procfile to Upstart configuration files

Our application has now be deployed to our server, but we need to initialize the processes and keep them monitored. Some Linux distributions come with Upstart pre-installed. If yours does not, you can install it from source by downloading it from the official website.

Assuming your machine does have Upstart installed, we need to write configuration files to the init directory. On Ubuntu, by default, this would mean writing the configuration files to /etc/init. It also requires you to have root access. If you’re using Foreman, you’ve actually already indirectly written the configuration files: your Procfile in your application’s root. Now we need to export (convert) that Procfile to Upstart configuration files.

You can do so by running the following command:

foreman export upstart /etc/init -a myapp

You should then see the following if all went well:

[foreman export] writing: /etc/init/myapp.conf
[foreman export] writing: /etc/init/myapp-web.conf
[foreman export] writing: /etc/init/myapp-web-1.conf
[foreman export] writing: /etc/init/myapp-worker.conf
[foreman export] writing: /etc/init/myapp-worker-1.conf
[foreman export] writing: /etc/init/myapp-pubsub.conf
[foreman export] writing: /etc/init/myapp-pubsub-1.conf

If you’re running this command as a non-root user, then you must prefix the command with sudo

sudo foreman export upstart /etc/init -a myapp

If it says it cannot find the foreman command, and you’re using RVM (Ruby Version Manager), be sure to use rvmsudo instead of just sudo

rvmsudo foreman export upstart /etc/init -a myapp

If you want to have more concurrency for some processes, you must specify it during export as well:

foreman export upstart /etc/init -a myapp -c web=3 worker=2 pubsub=1

Be sure to export it for the same user that’s deploying your application and has write access to your application path. If your deployments user name is “deployer”, you should specify it like so:

foreman export upstart /etc/init -a myapp -u deployer

If you don’t set it to the deployment user, it might be unable to properly start up your application. You can overwrite the configuration files at any time by re-running the foreman export command.

Here’s a full example of I’m using for one of my applications:

rvmsudo foreman export upstart /etc/init -a myapp -u deployer -c web=3 faye=1 worker=1

Start monitoring all your processes

Now that the configuration files are in place, all that’s left is to start the monitor and it’ll ensure your processes are always up and running, and restore them from crashes if necessary. To tell Upstart to monitor our processes, simply run the following command:

sudo start myapp

myapp is the name of your application you set by specifying the -a option in the foreman export command. If you do not specify the -a option, it’ll name your application after the name of the root directory where your application resides.

That’s it. Now all your processes should be up and running! Try to kill off a process using kill <pid> and you’ll see that it’ll be instantly respawned by the Upstart utility!

If you want Upstart to automatically start up and monitor “myapp” when you restart your server, you can add this line at the top of your /etc/init/myapp.conf file

start on runlevel [2345]

With this in place, each time you reboot your server, it’ll automatically spin up and monitor your application processes by itself so you don’t have to manually SSH in to do this.

Additionally you can also bind services other than your application-specific processes to Upstart via your Procfile, for example: nginx, mysql, postgresql, mongodb, redis, memcached, and so forth. You could also let Upstart handle that for you to ensure each of these is also always up and running, and starts up when your system boots. For example, let’s say we have Unicorn as our app server, Resque as our worker library, NGINX as our web server and MongoDB as our database. We would set up our Procfile to look something like this:

web:    bundle exec unicorn -p $PORT -E production
worker: bundle exec rake resque:work -e production
mongo:  sudo /etc/mongodb/bin/mongod --journal
nginx:  sudo /usr/local/sbin/nginx -c /etc/nginx/conf/nginx.conf

With that you can then re-export that to Upstart and run restart myapp. Upstart will now also start up and monitor your MongoDB and NGINX instances. Some of the commands that become available when exporting to Upstart are:

  • start myapp
  • stop myapp
  • restart myapp

These will start/stop/restart all processes (web, worker, mongo, nginx). If you only want to restart all the web (Thin, Unicorn) processes, because you just pushed changes to your application and need to restart the app instances for them to take effect:

  • restart myapp-web

These commands are available from anywhere on your machine, you don’t need to be in the root of your application or anything, but you’ll have to execute them as the root user, or with sudo.

As you can see, Foreman beautifully integrates in both your development, as well as your production environment. If you’ve set it up in your development environment, which should take less than a minute, you’re pretty much good to go when you deploy to production. I actually use 2 Procfiles: Procfile and Procfile.production. This is because I want to specify different flags for my production environment, such as -e production. You can tell Foreman to load a different Procfile, than the default.

foreman start                           # defaults to ./Procfile
foreman start -f ./Procfile.production  # uses ./Procfile.production

Conclusion

Foreman and Upstart can really save a lot of time, both in development, as well as in production. It’s a simple, reliable and awesome solution. It takes about a minute or less to set up locally for in development, and even less in production since all you have to do is export your Procfile and tell Upstart to run it. You have the joy of booting up your development environment with a single command to get to work quickly without having to remember all the command options/flags, and you can export your Procfile to Upstart in production and have it take care of keeping your processes up and running.

To Archive → Share twitter

Michael van Rooijen

Crafter of Ruby apps, and open source contributor.

I built and run HireFire.