We recently undertook an audit of a Rails deployment for a system developed by Red Comet Labs. It is a Rails 3.2 application that integrates with 3 different storage systems (CouchDB, PostgreSQL and Redis). It is deployed using Capistrano and the server setup and management is done using Chef.
We shared our Rails Deployment experience with Red Comet Labs, and in this article we’ll discuss a checklist of some devops practices and tools we are familiar with.
Extract sensitive data
All sensitive data like passwords, API keys & tokens that are in the application are extracted as environment variables outside of the source code repository. We are using rbevn for managing rubies, there is a nice plugin for setting up environment variablesrbenv-vars.
Continuous Integration & Deployment
There are many great open source and commercial tools out there for Continuous Integration. We are using Jenkins CI to run our test suite and handle Continuous Deployment (if the tests are green) for our staging branch. For production releases, we suggest doing releases manually at a scheduled time with the whole team ready and available.
If you need more advanced tool for doing the deploy, Capistrano is the definitely the tool to go with. However, if you want something more light-weight you can try gitploy (bare minimum, git-based tool for deployment). Alternatively, check out another tool from this list.
Document server setup and everything about the deployment process. It may seem tedious now but in the long-run it will help because setups are easily forgetten and the person who initially setup the server is not always on hand to assist. By using an automation tool like Chef or Puppet you are kind of documenting things on the fly.
Don’t use your root user when you access your server – it’s the simplest way to prevent accidentally breaking things. Create another user that can execute superuser commands using sudo. Use this non-root user all the time. You should use SSH key authentication to protect your server against brute-force password cracking attacks. Furthermore, you should disable SSH password authentication and the ability to login as root. Read more on this in Linode’s guide on how to secure your server.
Start clean on boot
All the services that are being used by the application need to start cleanly when the server boots up – there should be no need for manual intervention. If you are deploying to Ubuntu, update-rc.d can be used for init.d processes. And, for the application Procfile-based processes you can use the foreman gem and there is an easy export to upstart for monitoring and boot start up setup.
Prevent log files from growing to the point where you don’t have any space left on disc device by using logrotate. Analyze request log of your Rails application to produce a performance report with request-log-analyzer.
Whether it’s a backup script or background task that needs to be run as a job, it’s a good idea to have documentation on it within the application. Cron’s scheduling format is often difficult to read so we suggest using the whenever gem to make it more readable.
For both database & uploaded content (like images and documents) we need to have regular backups distributed in different physical or cloud locations, backup gem is very handy tool for that.
If server crashes, can you fail over to another server? How much time do you need for that operation and have you tested the procedure before it actually happens? Hosting services offer daily/weekly/monthly node backups but the procedure need to be tested before there is a need for doing that.
External monitoring tools
Notifications for Exceptions
Process monitoring tools
Monitor your application performance, uptime, slow DB queries, background tasks with New Relic. They also have Availability monitoring if you don’t want to use other service for checking that your server is up and running.
The list is long, but it takes lots of patience in understanding the software business and delivering great software.