How to setup a dev environment using Vagrant & Puppet (Part I)

Vagrant

Vagrant is a great tool for spinning up different dev environments without polluting your own local machine with gems and packages from different projects. It is also great for sharing the same environment across your entire team without having to going through the pain of manual setup every time you on-board a new team member.

You can use providers/platforms such as VMWare, VirtualBox, AWS, etc to run your VM. If you are a first-time user, installing VirtualBox is probably your best option to get started.

After you’ve installed VirtualBox and Vagrant, let’s create a folder where you want to initializing a base machine.

$ mkdir my-project-dev-box
$ cd my-project-dev-box
$ git init
$ vagrant init precise32 http://files.vagrantup.com/precise32.box

Those commands will create a plain Ubuntu 12.04 box and generate a Vagrantfile in my-project-box folder. It will also initialize a git repository since we’ll need that later down the line to add modules.

But the first thing we’ll do now is add custom hostname in the config block to avoid getting confused if you spin up more Vagrant machines in the future. I like to have the project’s name in the hostname so you can easily identify it (same as the folder name):

config.vm.hostname = 'some-project-dev-box'

Since we’ll be using this for a particular dev environment, we need more than just a plain Ubuntu box. So let’s look at how we’d go about installing other packages and software.

Vagrant supports many different provisioners/IT automation tools such as Ansible, Chef, Puppet, or even a plain old shell script to help us automate this process. For the purpose of this post, we’ll use Puppet since IMHO it has the easiest learning curve.

Puppet

To use Puppet for provisioning, add this block in the Vagrantfile:

config.vm.provision :puppet do |puppet|
    puppet.manifests_path = 'puppet/manifests'
    puppet.module_path    = 'puppet/modules'
    puppet.options        = '--verbose'
end

So as you might have guessed, we’ll need a puppet directory with two sub-directories manifests and modules. Manifests will contain a default.pp file which tell Puppet which modules to install and how:

|   .Vagrantfile
\---puppet
    +---manifests
    |       default.pp
    \---modules

Modules are basically just re-usable instructions for how to install specific piece of software. You can create your own modules or just grab some 3rd party modules from the Example42’s repository, Puppet Lab’s official repository, or just from other folks on GitHub.

There are basically three ways of installing/using these modules. You can either clone or just copy/paste the module in the modules folder, you can add it as a git submodule or use the puppet module tool. I’ve never tried the last one since it doesn’t work on Windows.

This is where I’ll stop for now. In my next post, I’ll talk about installing specific modules and how to overcome some common gotchas while configuring them. Meanwhile, here’s what our basic Vagrantfile looks like at this point:

Vagrant.configure('2') do |config|
  config.vm.box      = 'precise32'
  config.vm.box_url  = 'http://files.vagrantup.com/precise32.box'
  config.vm.hostname = 'some-project-dev-box'

  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = 'puppet/manifests'
    puppet.module_path    = 'puppet/modules'
  end
end

If you wanted to you can boot your VM and ssh into it just like any other Ubuntu machine:

$ vagrant up
$ vagrant ssh

Although at this point, there’s not much to see. We’ll get to the fun stuff in part II. Stay tuned!

Useful tools for web development

This time I’m trying something different. I want to talk about some dev tools that I find indispensable in my day-to-day coding tasks:

Vagrant - Helps you spin up different VMs to create isolated dev environments. Definitely worth checking out if you find yourself juggling many different projects at once. (Checkout my two part blog post on how to get started.)

ngrok - Ever found yourself needing to test your local app with external callbacks such as Facebook or Twitter auth? Well this lets you securely expose your localhost to internet traffic to do just that. No need to constantly deploy your app to a staging environment just to test callbacks.

RequestBin - Opposite of ngrok. It lets you quickly test and inspect webhooks or your outgoing HTTP requests to make sure they are formatted correctly before you deploy them to production.

Postman - One of the best Chrome extensions out there for testing and interacting with REST APIs.

ExpanDrive - Elegant solution to mount an external VPS or S3 to your local filesystem. It’s not free but it’s worth it.

HeidiSQL - Most lightweight and fastest SQL client out there. It gets the job done without a lot of fuss.

Notepad++ - I use different IDEs for various projects ranging from Visual Studio to RubyMine but this has been a constant companion for doing quick text manipulations or editing small chunks of code without opening up clunky IDEs.

Awesome Cookie Manager - When testing an app, no need to log out. Just delete the cookie from your browser itself or change it to a certain value to see if your app is responding to it correctly.

smtp4dev or FakeSMTP - These two will help you test emails in your web app without actually sending them out.

JSFiddle - No introduction needed here. If you’ve been developing for a while I’m sure you’ve come across this. Best way to demo your code snippet or get some help fixing a bug.

htacess tester - Name says it all. Quick and easy way to test your .htaccess files to make sure they are working as intended.

RegExr - Beautifully designed online tool to learn, build, & test Regular Expressions with a rich feature set including real time results, sharing, saving, examples library, etc

StackEdit - One of the best markdown editors out there with live previews. Very handy for writing readme files or wikis on GitHub.

explainshell - Found a command but not quite sure what it does? Before executing it, run it through explain shell to make sure it’s doing what you intended it to do.

I would love to hear from you. What are some of your favorite dev tools?

Automated deployments with git-flow, heroku_san & CircleCI

Here’s a simple workflow that I like to use to quickly get automated deployments working on any RoR project running on Heroku.

git-flow

First we need to establish a good git branching process and for that I always use git-flow.

If you are not familiar with it, I highly recommend reading A successful Git branching model which explains what it’s all about.

Basically it’s just a command line wrapper for several git commands. For example, this git-flow command git flow feature start cool_stuff will create a new branch called feature/cool_stuff, based on your develop branch and switch to it.

It’s super easy to use. It establishes best-practices out the door and helps you leverage the power of git.

heroku_san

Now once you have that setup, we can move on to setting up heroku_san. It makes maintaining multiple environments (staging, production, demo) a breeze by providing a simple set of rake tasks.

Install the gem

group :development do
    gem 'heroku_san'
end

and run the rake command

rails generate heroku_san

which will generate a /config/heroku.yml. You can tweak this according to your needs. In our case:

production:
  app: liveapp
  stack: cedar
  config:
    RACK_ENV: production
    RAILS_ENV: production
    BUNDLE_WITHOUT: "development:test"

staging:
  app: stagingapp
  stack: cedar
  config:
    RACK_ENV: staging
    RAILS_ENV: staging
    BUNDLE_WITHOUT: "development:test"

Now you could run rake staging deploy at anytime to deploy your code to the stagingapp on Heroku but we want to automate that.

CircleCI

That brings us to the last part. Here you could swap CircleCI with any other CI server such as Jenkins, Codeship, Travis, Semaphore, etc they should all work similarly.

In this case, go ahead and sign-up for CircleCI and add your project to it. After that, follow this guide to deploy to Heroku.

Once you’ve set that up, create a circle.yml file at the root of your project so it knows what to do when you push your code:

## Customize deployment commands
deployment:
  production:
    branch: master
    commands:
      - bundle exec rake production deploy
  staging:
    branch: develop
    commands:
      - bundle exec rake staging deploy

We are basically telling it to run heroku_san deploy tasks once your test suite passes.

In Action

So now that we’ve got it all dialed in, we can either push to develop or master and the code will be deployed to the correct app on Heroku.

Pushing to staging:

git push origin develop

That’s all it takes and your app will automatically be deployed to your stagingapp on Heroku. It will recognize that we pushed to the develop branch based on our configuration in the circle.yml.

OR

Pushing to production:

git flow release start 1.0.0
git flow release finish 1.0.0
git push origin master develop --tags

In this case, git-flow will create a release branch/tag which will be merged into master and pushed to GitHub. CI will listen in on that and immediately trigger a production deploy since we pushed to master.

Using Postgres Hstore with Rails 4

I’m a huge fan of PostgresSQL given it’s performance and stability. One of it’s great features is the ability to store hashes in tables using the Hstore column type AND to query them! This is perfect for a column like user settings where you may need to add or change the type of settings you need as your app evolves. Here’s some steps and gotchas that I came across while using this column.

Setup

So first let’s start by creating a migration which would enable the Hstore extension on our database:

class AddHstore < ActiveRecord::Migration
  def up
    execute 'CREATE EXTENSION hstore'
  end

  def down
    execute 'DROP EXTENSION hstore'
  end
end

Next you’ll need to change your schema dump format from ruby to sql as the schema for hstore can’t be represented by ruby.

In config/application.rb:

config.active_record.schema_format = :sql

I’ve also come across a more cleaner way to enable this without using the native execute command (although I never got it to work):

class SetupHstore < ActiveRecord::Migration
  def self.up
    enable_extension "hstore"
  end
  def self.down
    disable_extension "hstore"
  end
end

With this approach you don’t have to even change the schema_format, it will automatically add enable_extension "hstore" in your schema.rb file. Again, I’ve yet to see this work but I’d try this first before using the native sql approach.

Usage

Now you are ready to start using this column. Let’s add a settings column to our users table.

class AddSettingsToUsers < ActiveRecord::Migration
  def change
     add_column :users, :settings, :hstore  
  end
end

If you know which keys you’ll store in the settings column, you can define accessors in your model and even validate them:

class User < ActiveRecord::Base
  store_accessor :settings, :theme, :language

  validates :theme, inclusion: { in: %w{white, red, blue} }
  validates :language, inclusion: { in: I18n.available_locales.map(&:to_s) }
end

Although, if you don’t know what kind of keys the client will send your controller, here’s a way to receive dynamic keys/value pairs using strong params:

def some_params
  params.require(:some).permit(:name, :title).tap do |whitelisted|
    whitelisted[:settings] = params[:some][:settings]
  end
end

We are now ready to query our users based on their settings:

rubyUser.where("settings -> 'language' = 'en'")

More query syntax can be found here in the hstore documentation.

Indexes

hstore provides index support for a few select operators such as @>, ?, ?& or ?|. So if you are planning to use any of those, it’s probably a good idea to add an index for better performance:

class AddIndexToUserSettings < ActiveRecord::Migration
  def up
    execute 'CREATE INDEX users_settings ON users USING gin(settings)'
  end

  def down
    execute 'DROP INDEX users_settings'
  end
end

Alternatively you can use the native DSL to accomplish the same thing:

class AddIndexToUserSettings < ActiveRecord::Migration
  def up
    add_index :users, [:settings], name: "users_gin_settings", using: :gin
  end

  def down
    remove_index :users, name: "users_gin_settings"
  end
end

I generally tend to use GIN index as the lookups are three times faster compared to GiST indexes, although they take three times longer to build. You can checkout their documentation to decide which is the best fit for your needs.

Heroku

If you are using Heroku, there is one small issue I ran across. As a consequence of switching to sql schema format, I kept getting this error when heroku ran db:migrate at the end of each deployment:

No such file or directory - pg_dump -i -s -x -O -f /app/db/structure.sql

It’s trying to dump a new structure.sql with the pg_dump command which is not present on Heroku’s environment. You really don’t need that in production anyway. So the way to get around it would be to just silent that task if it’s running in any environment other than development by adding this in your Rakefile:

Rake::Task["db:structure:dump"].clear unless Rails.env.development?

That’s a wrap! Let me know if I missed anything.

Fixing CPU spikes in Ubuntu 11.10

If you finding your CPU usage spiking in Ubuntu 11.10 it may very well be an issue caused by php5’s cron task.

To confirm, first use top to see if fuser is taking up the most amount of CPU and creating hundreds of zombie processes. If that’s the case, here’s the fix:

Open /etc/cron.d/php5 and you should see something like this:

09,39 *     * * *     root   [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete

Go ahead and comment that out and replace it with:

09,39 *     * * *     root   [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete

The difference being that the 11.10 version runs fuser for each PHP session file, thus using all the CPU when there are hundreds of sessions.

Save that and reboot the instance to regain normalcy.

All credit goes to grazer for suggesting this solution.