git reflog

This command will return something like:

902532b HEAD@{5}: commit: Doing some stuff on master now...wait where's my previous commits?
7f6c682 HEAD@{7}: checkout: moving from 3a311d8f746f903e15ce145a3e7683e0d53536fb to master
3a311d8 HEAD@{8}: commit: Added some more stuff
56f7716 HEAD@{9}: commit: Added some stuff
7f6c682 HEAD@{10}: checkout: moving from master to 7f6c6821c80baff91492af4909f1012876ef6249

56f7716 and 3a311d8 are the SHA identifiers of the commits that are floating around in the git void, attached to no branch, so I want to move them into the master branch. Starting with the first commit:

git checkout 56f7716

This will return output similar to:

Note: moving to '56f7716' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
git checkout -b
HEAD is now at 56f7716... Added some stuff

Pretty self explanatory. Let's do what it suggests and create a branch at this point in the history tree. We can then merge that branch into master (or whichever branch it belongs to).

git checkout -b temp_branch
git checkout master
git merge temp_branch

The commit is now in master where it belongs. Rinse and repeat for commit 3a311d8 and all is well in the world again!

My approach now is to keep the existing vendor/rails directory and copy the new Rails files into it, doing a recursive diff of the two directories first to identify files and directories that need to be deleted as they don't exist in the new version. The process is a little bit fiddly but I've had no merge issues since adopting it, so I find it worth the extra effort.

Here's my step-by-step process:

1 - Temporarily rename vendor/rails so the latest Rails version can be frozen (all commands are assuming you in the Rails root directory):

mv vendor/rails vendor/rails_old

2 - Freeze the newer version of Rails (in this case, 2.3.9 as the app isn't quite ready for a Rails 3 upgrade yet)

rake rails:freeze:gems VERSION=2.3.9

This creates vendor/rails with version 2.3.9 of the Rails gems.

3 - Move vendor/rails_old back to vendor/rails so the subversion tracking doesn't break.

mv vendor/rails vendor/rails_new
mv vendor/rails_old vendor/rails

Do a quick 'svn status' to confirm everything is correct.

svn st vendor

It should look like this:

? vendor/rails_new

4 - Now do a diff of the two Rails directories to see what files only exist in the old version of Rails. These are the files we want to delete.

diff -qr vendor/rails vendor/rails_new | grep "Only in vendor/rails/" | grep -v -e "svn" > rails_diffs

Note the use of grep to return only files that exist in rails/. The output is also written to ./rails_diffs as we need to edit the output slightly. It will look something like this:

Only in vendor/rails/actionmailer/lib/action_mailer/vendor: tmail-1.1.0
Only in vendor/rails/actionmailer/lib/action_mailer/vendor: tmail-1.2.3
Only in vendor/rails/actionmailer/lib/action_mailer: vendor.rb
Only in vendor/rails/actionmailer/test/fixtures/first_mailer: share.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_example_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_helper_method.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_mail_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/path.with.dots/funky_path_mailer: multipart_with_template_path_with_dots.rhtml
Only in vendor/rails/actionmailer/test/fixtures/path.with.dots: multipart_with_template_path_with_dots.rhtml
Only in vendor/rails/actionmailer/test/fixtures: raw_base64_decoded_string
Only in vendor/rails/actionmailer/test/fixtures: raw_base64_encoded_string
Only in vendor/rails/actionmailer/test/fixtures/second_mailer: share.rhtml
Only in vendor/rails/actionmailer/test/fixtures/templates: signed_up.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.ignored.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.html.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.plain.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.yaml.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up.erb
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up_with_url.rhtml
Only in vendor/rails/actionpack/lib/action_controller: assertions.rb
Only in vendor/rails/actionpack/lib/action_controller/caching: sql_cache.rb
Only in vendor/rails/actionpack/lib/action_controller/cgi_ext: session.rb
Only in vendor/rails/actionpack/lib/action_controller: components.rb
Only in vendor/rails/actionpack/lib/action_controller: request_profiler.rb
Only in vendor/rails/actionpack/lib/action_controller: routing_optimisation.rb
Only in vendor/rails/actionpack/lib/action_controller/session: active_record_store.rb
Only in vendor/rails/actionpack/lib/action_controller/session: drb_server.rb
Only in vendor/rails/actionpack/lib/action_controller/session: drb_store.rb
Only in vendor/rails/actionpack/lib/action_view: compiled_templates.rb
Only in vendor/rails/actionpack/lib/action_view/helpers: javascripts
Only in vendor/rails/actionpack/lib/action_view: partial_template.rb
Only in vendor/rails/actionpack/lib/action_view: template_finder.rb
Only in vendor/rails/actionpack/lib/action_view/template_handlers: compilable.rb
Only in vendor/rails/actionpack/test: action_view_test.rb
Only in vendor/rails/actionpack/test/controller: cgi_test.rb
Only in vendor/rails/actionpack/test/controller: components_test.rb
Only in vendor/rails/actionpack/test/controller: custom_handler_test.rb
Only in vendor/rails/actionpack/test/controller: fragment_store_setting_test.rb
Only in vendor/rails/actionpack/test/controller: http_authentication_test.rb
Only in vendor/rails/actionpack/test/controller: integration_upload_test.rb
Only in vendor/rails/actionpack/test/controller: new_render_test.rb
Only in vendor/rails/actionpack/test/controller: session_fixation_test.rb
Only in vendor/rails/actionpack/test/controller: session_management_test.rb
Only in vendor/rails/actionpack/test/fixtures/test: block_content_for.erb
Only in vendor/rails/actionpack/test/fixtures/test: erb_content_for.erb
Only in vendor/rails/actionpack/test/template: deprecated_erb_variable_test.rb
Only in vendor/rails/actionpack/test/template: template_finder_test.rb
Only in vendor/rails/actionpack/test/template: template_object_test.rb
Only in vendor/rails/activerecord/lib/active_record: vendor
Only in vendor/rails/activerecord/test: aaa_create_tables_test.rb
Only in vendor/rails/activerecord/test: abstract_unit.rb
Only in vendor/rails/activerecord/test: active_schema_test_mysql.rb


5 - All the files we want to delete are in rails_diffs, so using your favourite text editor do a I find of all instances of "Only in", replacing it with "svn rm" and also find ": " (note single white space) and replace it with "/". This will modify the file to look something like:

svn rm vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.1.0
svn rm vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.2.3
svn rm vendor/rails/actionmailer/lib/action_mailer/vendor.rb
svn rm vendor/rails/actionmailer/test/fixtures/first_mailer/share.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_example_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_helper_method.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_mail_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/path.with.dots/funky_path_mailer/multipart_with_template_path_with_dots.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/path.with.dots/multipart_with_template_path_with_dots.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/raw_base64_decoded_string
svn rm vendor/rails/actionmailer/test/fixtures/raw_base64_encoded_string
svn rm vendor/rails/actionmailer/test/fixtures/second_mailer/share.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/templates/signed_up.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.ignored.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.html.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.plain.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.yaml.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.erb
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up_with_url.rhtml
svn rm vendor/rails/actionpack/lib/action_controller/assertions.rb
svn rm vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb
svn rm vendor/rails/actionpack/lib/action_controller/cgi_ext/session.rb
svn rm vendor/rails/actionpack/lib/action_controller/components.rb
svn rm vendor/rails/actionpack/lib/action_controller/request_profiler.rb
svn rm vendor/rails/actionpack/lib/action_controller/routing_optimisation.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/active_record_store.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/drb_server.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/drb_store.rb
svn rm vendor/rails/actionpack/lib/action_view/compiled_templates.rb
svn rm vendor/rails/actionpack/lib/action_view/helpers/javascripts
svn rm vendor/rails/actionpack/lib/action_view/partial_template.rb
svn rm vendor/rails/actionpack/lib/action_view/template_finder.rb
svn rm vendor/rails/actionpack/lib/action_view/template_handlers/compilable.rb
svn rm vendor/rails/actionpack/test/action_view_test.rb
svn rm vendor/rails/actionpack/test/controller/cgi_test.rb
svn rm vendor/rails/actionpack/test/controller/components_test.rb
svn rm vendor/rails/actionpack/test/controller/custom_handler_test.rb
svn rm vendor/rails/actionpack/test/controller/fragment_store_setting_test.rb
svn rm vendor/rails/actionpack/test/controller/http_authentication_test.rb
svn rm vendor/rails/actionpack/test/controller/integration_upload_test.rb
svn rm vendor/rails/actionpack/test/controller/new_render_test.rb
svn rm vendor/rails/actionpack/test/controller/session_fixation_test.rb
svn rm vendor/rails/actionpack/test/controller/session_management_test.rb
svn rm vendor/rails/actionpack/test/fixtures/test/block_content_for.erb
svn rm vendor/rails/actionpack/test/fixtures/test/erb_content_for.erb
svn rm vendor/rails/actionpack/test/template/deprecated_erb_variable_test.rb
svn rm vendor/rails/actionpack/test/template/template_finder_test.rb
svn rm vendor/rails/actionpack/test/template/template_object_test.rb
svn rm vendor/rails/activerecord/lib/active_record/vendor
svn rm vendor/rails/activerecord/test/aaa_create_tables_test.rb
svn rm vendor/rails/activerecord/test/abstract_unit.rb
svn rm vendor/rails/activerecord/test/active_schema_test_mysql.rb


This creates a list of commands to run to remove those files/directories from version control. So copy the contents of rails_diff and paste it into your command line. You'll see a bunch of output confirming Subversion has removed those files/directories.

6 - While "svn rm" will physically delete files, directories still remain, so lets tidy those up to. Go back to rails_diff and find all instances of "svn rm", replace with "rm -rf" and again copy and paste the contents of the file into the command line. Yes this will try and delete files that have already been deleted but as we're forcing the delete with the '-f' option, it won't complain.

7 - Now that all the unneeded files from the old version of Rails have been removed it's time to update vendor/rails/ to the our newer version.

cp -R vendor/rails_new/ vendor/rails/

This will modify existing files that have changed in the newer version of Rails and add any new files. Modified files are already tracked by Subversion, but we need to add the new files to version control. So in a similar way to how we diff'd the two rails directories, we'll use 'svn status' to tell us what files are new.

svn st vendor/rails | grep ? > rails_diffs

The output again goes into ./rails_diff and will look something like:

? vendor/rails/activeresource/test/debug.log
? vendor/rails/activeresource/test/http_mock_test.rb
? vendor/rails/activeresource/test/fixtures/proxy.rb
? vendor/rails/activeresource/lib/active_resource/exceptions.rb
? vendor/rails/actionmailer/test/mail_layout_test.rb
? vendor/rails/actionmailer/test/asset_host_test.rb
? vendor/rails/actionmailer/test/fixtures/auto_layout_mailer
? vendor/rails/actionmailer/test/fixtures/layouts
? vendor/rails/actionmailer/test/fixtures/asset_host_mailer
? vendor/rails/actionmailer/test/fixtures/explicit_layout_mailer
? vendor/rails/actionmailer/test/fixtures/test_mailer/body_ivar.erb
? vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.html.erb
? vendor/rails/actionmailer/lib/action_mailer/vendor/tmail.rb
? vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.2.7
? vendor/rails/actionmailer/lib/action_mailer/vendor/text_format.rb
? vendor/rails/activesupport/lib/active_support/message_verifier.rb
? vendor/rails/activesupport/lib/active_support/memoizable.rb
? vendor/rails/activesupport/lib/active_support/secure_random.rb
? vendor/rails/activesupport/lib/active_support/message_encryptor.rb
? vendor/rails/activesupport/lib/active_support/locale
? vendor/rails/activesupport/lib/active_support/rescuable.rb
? vendor/rails/activesupport/lib/active_support/all.rb
? vendor/rails/activesupport/lib/active_support/xml_mini.rb

8 - Do a find of '?' and replace it with 'svn add'. Again, copy and paste the file contents into the command line. This will add all these files to version control. Once complete, check the svn status of vendor/rails again to confirm that everything is now tracked.

svn st vendor/rails | grep ?

If this outputs nothing then you're good to go.

9 - Your vendor/rails directory is now up to date and ready to commit to your repository.

svn commit vendor/rails -m "Upgraded vendor/rails from vX.X.X to vX.X.X"

10 - Finally a bit of tidying up:

rm -rf vendor/rails_new
rm ./rails_diffs

All done!

I recently had to change Virtual Private Server provider and made the switch to the excellent Linode. A colleague recommended them and I've been impressed with their service, cost, features and reliability.

While making the switch I took the opportunity to document the steps I took to setup a fresh VPS, primarily for hosting Ruby on Rails, but also as a Git server and Subversion server. Here are those steps, for my own future reference, but if others find it useful, all well and good...

Initial setup

Linode comes with a number of pre-configured "StackScripts" which automates setting up of various software on your server when it boots for the first time. As my server would be hosting Ruby on Rails applications, I chose the "Apache, MySQL, Ruby Setup" Stackscript, on Debian 5.

After the initial boot, Apache, Mysql and Ruby were indeed installed by the script. However, I noticed the Ruby Gems installation did not complete, so I did it manually as follows:

wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz
tar xzvf rubygems-1.3.6.tgz
cd rubygems-1.3.6
ruby setup.rb

This installs the 'gem' binary to /usr/local/bin, so no need to update $PATH.

I won't go into detail about the various gems I installed, as apart from the obvious gems, these vary depending on your application(s) requirements. However, while installing gems, I did encounter problems with the mysql gem failing to compile as it required the libmysqlclient-dev package:

apt-get install libmysqlclient-dev
gem install mysql -- --with-mysql-dir=/usr/bin/mysql_config

Hello Joe!

Before I continued with anything else, I installed my first and favourite Linux text editor - Joe!

apt-get install joe

Joe is quite paranoid and automatically creates backups of all modified files. This litters the filesystem with lots of files with names ending in ~, which gets irritating, fast. I disable auto-backups like so:

edit /etc/joe/joerc
Find line containing 'Default local options'
add under it '-nobackups'
save

Git

Next up, version control, starting with Git.

apt-get install git-core

Running a private Git server using gitosis takes a little more work, so I'll point you in the direction of Garry Dolley's excellent tutorial.

I did note while following this tutorial that python-setuptools had to be installed:

apt-get install python-setuptools

Subversion

I have a number of older projects which I have not yet ported to Subversion, so better install that too right?

apt-get install subversion

I like being able to browse my SVN repository in my browser, so I updated Apache with mod-DAV.

apt-get install libapache2-svn
a2enmod dav

I don't want people snooping my source code, so next up was enabling SSL in Apache:

a2enmod ssl

I then created a new virtualhost for browsing the repository, with SSL and HTTP authentication:

<VirtualHost *:443>
  ServerName my.host.name

  SSLEngine on
  SSLCertificateFile /path/to/certificate/file
  SSLCertificateKeyFile /path/to/certificate/key/file
  SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown

  <Location />
    DAV svn
    SVNPath /path/to/svn/repository/
    AuthType Basic
    AuthName "Restricted Access"
    AuthUserFile /path/to/password/file
    Require valid-user
  </Location>

  ErrorLog /path/to/error/log
  CustomLog /path/to/access/log combined
</VirtualHost>

Note, there are still further steps here to setup the SSL certificate (I used self-signed certs) and the HTTP authentication password file. But I'll not go into detail on that, a quick Google search will reveal plenty of guides to doing both.

Nginx

Nginx is a slim and snappy web server, great for serving up static files super fast and acting as a reverse proxy to Rails application servers (e.g. Mongrel/Thin/Unicorn). While I often just stick with Apache as a web server, I've been experimenting with Nginx and so installed from source as follows:

Note: For rewrite in Nginx, the PCRE library had to be installed (See http://www.cyberciti.biz/faq/debian-ubuntu-linux-install-libpcre3-dev/):

apt-get install libpcre3 libpcre3-dev

Download the latest stable source code from http://wiki.nginx.org/NginxInstall, extract, configure and compile.

tar xzfv nginx-0.8.53.tar.gz
cd nginx-0.8.53
./configure --with-http_ssl_module
make
make install

Usergroups

Simple system administration stuff, but I like to have a staff usergroup where I put all my users. This allows for assigning permissions more easily to this group, and in turn to those users (see the following section on SSH):

addgroup <usergroupname>
adduser <username> --ingroup <usergroupname>

Lock it down!

SSH is the doorway into my server, so I like to keep it locked pretty tight. The following are a few tweaks I make to the default SSH Daemon configuration /etc/ssh/sshd_config:

Change: LoginGraceTime 120
To: LoginGraceTime 40

Change: PermitRootLogin yes
To: PermitRootLogin no

Disable password authentication in favour of RSA authentication (read more on setting this up).

Change: PasswordAuthentication yes
To: PasswordAuthentication no

Under "Authentication:" add:

MaxAuthTries 3

If a staff group exists and it has existing users, add at the end of the file:

AllowGroups staff

Finally, after saving those changes, reload the SSH daemon so they take effect (double check all your changes first, as you might lock yourself out if you made a mistake):

/etc/init.d/ssh reload

Rainer Wichmann's write-up at http://www.la-samhna.de/library/brutessh.html is well worth reading for more SSH security tips.

Monitoring

Monitoring your critical services is a must for any production system, and as far as I'm concerned you need look no further than monit. I've played around with God and while writing your monitoring configuration with Ruby is nice, I found it unstable, buggy and the memory footprint far too large, as have others. Monit is proven, stable, lightweight and configuring it is a sintch.

apt-get install monit

ImageMagick & Friends

If you do any kind of image manipulation in your Rails apps you'll inevitably need ImageMagick and RMagick (or mini_magick, ImageScience etc).

Installing these can be a real pain (see OS X!), but thankfully I found it straightforward on Debian 5, with one gotcha, I had to use rmagick v2.12.2 as later versions refused to compile.

apt-get install imagemagick
apt-get install libmagick9-dev
gem install rmagick --no-ri --no-rdoc --version=2.12.2
gem install mini_magick --no-ri --no-rdoc

Once all installed, remove libmagick9-dev and its dependencies as they are no longer needed:

apt-get remove libmagick9-dev
apt-get autoremove

Other bits and bobs

The Wordpress Google Reader module I use on this site requires the PHP Curl module, installed as follows:

apt-get install php5-curl

Wordpress also needs mod-rewrite for pretty URLs:

a2enmod rewrite

I also tend not to use sudo, instead preferring to switch user to root after I SSH into the server, so I usually remove it:

apt-get remove sudo

And don't forget to update your server's hostname:

hostname desired-host-name

Keeping things tidy

Finally, a little bit of house cleaning to remove any cruft left behind by Aptitude. Who knows, you might need the extra disc space one day!

apt-get clean

And we're done!

The power of Wordpress never ceases to amaze me.  In little over an hour, I've gone from an empty htdocs/ directory to a fully functional website with integrated Google Reader, Twitter and Github project feeds. Not bad.

I toyed with the idea of rolling my own new version of Abecto.com, but having played with Wordpress on RedCafe.net for the Blog and Podcast sections, I knew I could get 90% of the features I wanted through it and so here we are.

The current theme is Lightword by Andrei Luca, one that caught my eye after some brief browsing of the available Themes from wp-themes.com.  I'll probably reskin with my own theme eventually, but for now, this is more than adequate.

This blog will be as much a resource for myself to remind me of things I have discovered and learnt, but too quickly forgotten.  If somebody somewhere out there finds it of value, all the better.

Enjoy your stay at Abecto.com v2!