Retrieving lost Git commits
From time to time I've managed to "lose" commits in Git, usually after I have initialised a submodule in a project and forgot to checkout a specific branch before committing changes. When I eventually do realise my mistake, I need to get those commits back into the relevant branch - in this case, master. Here's how it's done.
git reflog
This command will return something like:
902532b HEAD@{5}: commit: Doing some stuff on master now...wait where's my previous commits?
7f6c682 HEAD@{7}: checkout: moving from 3a311d8f746f903e15ce145a3e7683e0d53536fb to master
3a311d8 HEAD@{8}: commit: Added some more stuff
56f7716 HEAD@{9}: commit: Added some stuff
7f6c682 HEAD@{10}: checkout: moving from master to 7f6c6821c80baff91492af4909f1012876ef6249
56f7716 and 3a311d8 are the SHA identifiers of the commits that are floating around in the git void, attached to no branch, so I want to move them into the master branch. Starting with the first commit:
git checkout 56f7716
This will return output similar to:
Note: moving to '56f7716' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
git checkout -b
HEAD is now at 56f7716... Added some stuff
Pretty self explanatory. Let's do what it suggests and create a branch at this point in the history tree. We can then merge that branch into master (or whichever branch it belongs to).
git checkout -b temp_branch
git checkout master
git merge temp_branch
The commit is now in master where it belongs. Rinse and repeat for commit 3a311d8 and all is well in the world again!
Upgrading version controlled Rails gems frozen in vendor/rails
I recently had to upgrade quite an old Rails application from Rails v2.1 to v2.3.9. Previously I would do this by deleting the old rails/ directory in vendor/ and refreezing the gems. However, I have had the strange errors from Subversion when merging the upgrade branch back into trunk and it caused me a sufficient headache to find an alternative solution.
My approach now is to keep the existing vendor/rails directory and copy the new Rails files into it, doing a recursive diff of the two directories first to identify files and directories that need to be deleted as they don't exist in the new version. The process is a little bit fiddly but I've had no merge issues since adopting it, so I find it worth the extra effort.
Here's my step-by-step process:
1 - Temporarily rename vendor/rails so the latest Rails version can be frozen (all commands are assuming you in the Rails root directory):
mv vendor/rails vendor/rails_old
2 - Freeze the newer version of Rails (in this case, 2.3.9 as the app isn't quite ready for a Rails 3 upgrade yet)
rake rails:freeze:gems VERSION=2.3.9
This creates vendor/rails with version 2.3.9 of the Rails gems.
3 - Move vendor/rails_old back to vendor/rails so the subversion tracking doesn't break.
mv vendor/rails vendor/rails_new
mv vendor/rails_old vendor/rails
Do a quick 'svn status' to confirm everything is correct.
svn st vendor
It should look like this:
? vendor/rails_new
4 - Now do a diff of the two Rails directories to see what files only exist in the old version of Rails. These are the files we want to delete.
diff -qr vendor/rails vendor/rails_new | grep "Only in vendor/rails/" | grep -v -e "svn" > rails_diffs
Note the use of grep to return only files that exist in rails/. The output is also written to ./rails_diffs as we need to edit the output slightly. It will look something like this:
Only in vendor/rails/actionmailer/lib/action_mailer/vendor: tmail-1.1.0
Only in vendor/rails/actionmailer/lib/action_mailer/vendor: tmail-1.2.3
Only in vendor/rails/actionmailer/lib/action_mailer: vendor.rb
Only in vendor/rails/actionmailer/test/fixtures/first_mailer: share.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_example_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_helper_method.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_mail_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/path.with.dots/funky_path_mailer: multipart_with_template_path_with_dots.rhtml
Only in vendor/rails/actionmailer/test/fixtures/path.with.dots: multipart_with_template_path_with_dots.rhtml
Only in vendor/rails/actionmailer/test/fixtures: raw_base64_decoded_string
Only in vendor/rails/actionmailer/test/fixtures: raw_base64_encoded_string
Only in vendor/rails/actionmailer/test/fixtures/second_mailer: share.rhtml
Only in vendor/rails/actionmailer/test/fixtures/templates: signed_up.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.ignored.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.html.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.plain.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.yaml.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up.erb
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up_with_url.rhtml
Only in vendor/rails/actionpack/lib/action_controller: assertions.rb
Only in vendor/rails/actionpack/lib/action_controller/caching: sql_cache.rb
Only in vendor/rails/actionpack/lib/action_controller/cgi_ext: session.rb
Only in vendor/rails/actionpack/lib/action_controller: components.rb
Only in vendor/rails/actionpack/lib/action_controller: request_profiler.rb
Only in vendor/rails/actionpack/lib/action_controller: routing_optimisation.rb
Only in vendor/rails/actionpack/lib/action_controller/session: active_record_store.rb
Only in vendor/rails/actionpack/lib/action_controller/session: drb_server.rb
Only in vendor/rails/actionpack/lib/action_controller/session: drb_store.rb
Only in vendor/rails/actionpack/lib/action_view: compiled_templates.rb
Only in vendor/rails/actionpack/lib/action_view/helpers: javascripts
Only in vendor/rails/actionpack/lib/action_view: partial_template.rb
Only in vendor/rails/actionpack/lib/action_view: template_finder.rb
Only in vendor/rails/actionpack/lib/action_view/template_handlers: compilable.rb
Only in vendor/rails/actionpack/test: action_view_test.rb
Only in vendor/rails/actionpack/test/controller: cgi_test.rb
Only in vendor/rails/actionpack/test/controller: components_test.rb
Only in vendor/rails/actionpack/test/controller: custom_handler_test.rb
Only in vendor/rails/actionpack/test/controller: fragment_store_setting_test.rb
Only in vendor/rails/actionpack/test/controller: http_authentication_test.rb
Only in vendor/rails/actionpack/test/controller: integration_upload_test.rb
Only in vendor/rails/actionpack/test/controller: new_render_test.rb
Only in vendor/rails/actionpack/test/controller: session_fixation_test.rb
Only in vendor/rails/actionpack/test/controller: session_management_test.rb
Only in vendor/rails/actionpack/test/fixtures/test: block_content_for.erb
Only in vendor/rails/actionpack/test/fixtures/test: erb_content_for.erb
Only in vendor/rails/actionpack/test/template: deprecated_erb_variable_test.rb
Only in vendor/rails/actionpack/test/template: template_finder_test.rb
Only in vendor/rails/actionpack/test/template: template_object_test.rb
Only in vendor/rails/activerecord/lib/active_record: vendor
Only in vendor/rails/activerecord/test: aaa_create_tables_test.rb
Only in vendor/rails/activerecord/test: abstract_unit.rb
Only in vendor/rails/activerecord/test: active_schema_test_mysql.rb
5 - All the files we want to delete are in rails_diffs, so using your favourite text editor do a I find of all instances of "Only in", replacing it with "svn rm" and also find ": " (note single white space) and replace it with "/". This will modify the file to look something like:
svn rm vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.1.0
svn rm vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.2.3
svn rm vendor/rails/actionmailer/lib/action_mailer/vendor.rb
svn rm vendor/rails/actionmailer/test/fixtures/first_mailer/share.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_example_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_helper_method.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_mail_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/path.with.dots/funky_path_mailer/multipart_with_template_path_with_dots.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/path.with.dots/multipart_with_template_path_with_dots.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/raw_base64_decoded_string
svn rm vendor/rails/actionmailer/test/fixtures/raw_base64_encoded_string
svn rm vendor/rails/actionmailer/test/fixtures/second_mailer/share.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/templates/signed_up.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.ignored.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.html.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.plain.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.yaml.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.erb
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up_with_url.rhtml
svn rm vendor/rails/actionpack/lib/action_controller/assertions.rb
svn rm vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb
svn rm vendor/rails/actionpack/lib/action_controller/cgi_ext/session.rb
svn rm vendor/rails/actionpack/lib/action_controller/components.rb
svn rm vendor/rails/actionpack/lib/action_controller/request_profiler.rb
svn rm vendor/rails/actionpack/lib/action_controller/routing_optimisation.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/active_record_store.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/drb_server.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/drb_store.rb
svn rm vendor/rails/actionpack/lib/action_view/compiled_templates.rb
svn rm vendor/rails/actionpack/lib/action_view/helpers/javascripts
svn rm vendor/rails/actionpack/lib/action_view/partial_template.rb
svn rm vendor/rails/actionpack/lib/action_view/template_finder.rb
svn rm vendor/rails/actionpack/lib/action_view/template_handlers/compilable.rb
svn rm vendor/rails/actionpack/test/action_view_test.rb
svn rm vendor/rails/actionpack/test/controller/cgi_test.rb
svn rm vendor/rails/actionpack/test/controller/components_test.rb
svn rm vendor/rails/actionpack/test/controller/custom_handler_test.rb
svn rm vendor/rails/actionpack/test/controller/fragment_store_setting_test.rb
svn rm vendor/rails/actionpack/test/controller/http_authentication_test.rb
svn rm vendor/rails/actionpack/test/controller/integration_upload_test.rb
svn rm vendor/rails/actionpack/test/controller/new_render_test.rb
svn rm vendor/rails/actionpack/test/controller/session_fixation_test.rb
svn rm vendor/rails/actionpack/test/controller/session_management_test.rb
svn rm vendor/rails/actionpack/test/fixtures/test/block_content_for.erb
svn rm vendor/rails/actionpack/test/fixtures/test/erb_content_for.erb
svn rm vendor/rails/actionpack/test/template/deprecated_erb_variable_test.rb
svn rm vendor/rails/actionpack/test/template/template_finder_test.rb
svn rm vendor/rails/actionpack/test/template/template_object_test.rb
svn rm vendor/rails/activerecord/lib/active_record/vendor
svn rm vendor/rails/activerecord/test/aaa_create_tables_test.rb
svn rm vendor/rails/activerecord/test/abstract_unit.rb
svn rm vendor/rails/activerecord/test/active_schema_test_mysql.rb
This creates a list of commands to run to remove those files/directories from version control. So copy the contents of rails_diff and paste it into your command line. You'll see a bunch of output confirming Subversion has removed those files/directories.
6 - While "svn rm" will physically delete files, directories still remain, so lets tidy those up to. Go back to rails_diff and find all instances of "svn rm", replace with "rm -rf" and again copy and paste the contents of the file into the command line. Yes this will try and delete files that have already been deleted but as we're forcing the delete with the '-f' option, it won't complain.
7 - Now that all the unneeded files from the old version of Rails have been removed it's time to update vendor/rails/ to the our newer version.
cp -R vendor/rails_new/ vendor/rails/
This will modify existing files that have changed in the newer version of Rails and add any new files. Modified files are already tracked by Subversion, but we need to add the new files to version control. So in a similar way to how we diff'd the two rails directories, we'll use 'svn status' to tell us what files are new.
svn st vendor/rails | grep ? > rails_diffs
The output again goes into ./rails_diff and will look something like:
? vendor/rails/activeresource/test/debug.log
? vendor/rails/activeresource/test/http_mock_test.rb
? vendor/rails/activeresource/test/fixtures/proxy.rb
? vendor/rails/activeresource/lib/active_resource/exceptions.rb
? vendor/rails/actionmailer/test/mail_layout_test.rb
? vendor/rails/actionmailer/test/asset_host_test.rb
? vendor/rails/actionmailer/test/fixtures/auto_layout_mailer
? vendor/rails/actionmailer/test/fixtures/layouts
? vendor/rails/actionmailer/test/fixtures/asset_host_mailer
? vendor/rails/actionmailer/test/fixtures/explicit_layout_mailer
? vendor/rails/actionmailer/test/fixtures/test_mailer/body_ivar.erb
? vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.html.erb
? vendor/rails/actionmailer/lib/action_mailer/vendor/tmail.rb
? vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.2.7
? vendor/rails/actionmailer/lib/action_mailer/vendor/text_format.rb
? vendor/rails/activesupport/lib/active_support/message_verifier.rb
? vendor/rails/activesupport/lib/active_support/memoizable.rb
? vendor/rails/activesupport/lib/active_support/secure_random.rb
? vendor/rails/activesupport/lib/active_support/message_encryptor.rb
? vendor/rails/activesupport/lib/active_support/locale
? vendor/rails/activesupport/lib/active_support/rescuable.rb
? vendor/rails/activesupport/lib/active_support/all.rb
? vendor/rails/activesupport/lib/active_support/xml_mini.rb
8 - Do a find of '?' and replace it with 'svn add'. Again, copy and paste the file contents into the command line. This will add all these files to version control. Once complete, check the svn status of vendor/rails again to confirm that everything is now tracked.
svn st vendor/rails | grep ?
If this outputs nothing then you're good to go.
9 - Your vendor/rails directory is now up to date and ready to commit to your repository.
svn commit vendor/rails -m "Upgraded vendor/rails from vX.X.X to vX.X.X"
10 - Finally a bit of tidying up:
rm -rf vendor/rails_new
rm ./rails_diffs
All done!
Setting up a new VPS on Linode
I host most of my personal and freelance sites on Virtual Private Servers. Why not shared hosting? Primarily because of the word "private". A VPS gives you full control of your own server, and the freedom to configure it as you wish, which in my experience is a must when hosting Ruby on Rails applications.
I recently had to change Virtual Private Server provider and made the switch to the excellent Linode. A colleague recommended them and I've been impressed with their service, cost, features and reliability.
While making the switch I took the opportunity to document the steps I took to setup a fresh VPS, primarily for hosting Ruby on Rails, but also as a Git server and Subversion server. Here are those steps, for my own future reference, but if others find it useful, all well and good...
Initial setup
Linode comes with a number of pre-configured "StackScripts" which automates setting up of various software on your server when it boots for the first time. As my server would be hosting Ruby on Rails applications, I chose the "Apache, MySQL, Ruby Setup" Stackscript, on Debian 5.
After the initial boot, Apache, Mysql and Ruby were indeed installed by the script. However, I noticed the Ruby Gems installation did not complete, so I did it manually as follows:
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz
tar xzvf rubygems-1.3.6.tgz
cd rubygems-1.3.6
ruby setup.rb
This installs the 'gem' binary to /usr/local/bin, so no need to update $PATH.
I won't go into detail about the various gems I installed, as apart from the obvious gems, these vary depending on your application(s) requirements. However, while installing gems, I did encounter problems with the mysql gem failing to compile as it required the libmysqlclient-dev package:
apt-get install libmysqlclient-dev
gem install mysql -- --with-mysql-dir=/usr/bin/mysql_config
Hello Joe!
Before I continued with anything else, I installed my first and favourite Linux text editor - Joe!
apt-get install joe
Joe is quite paranoid and automatically creates backups of all modified files. This litters the filesystem with lots of files with names ending in ~, which gets irritating, fast. I disable auto-backups like so:
edit /etc/joe/joerc
Find line containing 'Default local options'
add under it '-nobackups'
save
Git
Next up, version control, starting with Git.
apt-get install git-core
Running a private Git server using gitosis takes a little more work, so I'll point you in the direction of Garry Dolley's excellent tutorial.
I did note while following this tutorial that python-setuptools had to be installed:
apt-get install python-setuptools
Subversion
I have a number of older projects which I have not yet ported to Subversion, so better install that too right?
apt-get install subversion
I like being able to browse my SVN repository in my browser, so I updated Apache with mod-DAV.
apt-get install libapache2-svn
a2enmod dav
I don't want people snooping my source code, so next up was enabling SSL in Apache:
a2enmod ssl
I then created a new virtualhost for browsing the repository, with SSL and HTTP authentication:
<VirtualHost *:443>
ServerName my.host.name
SSLEngine on
SSLCertificateFile /path/to/certificate/file
SSLCertificateKeyFile /path/to/certificate/key/file
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
<Location />
DAV svn
SVNPath /path/to/svn/repository/
AuthType Basic
AuthName "Restricted Access"
AuthUserFile /path/to/password/file
Require valid-user
</Location>
ErrorLog /path/to/error/log
CustomLog /path/to/access/log combined
</VirtualHost>Note, there are still further steps here to setup the SSL certificate (I used self-signed certs) and the HTTP authentication password file. But I'll not go into detail on that, a quick Google search will reveal plenty of guides to doing both.
Nginx
Nginx is a slim and snappy web server, great for serving up static files super fast and acting as a reverse proxy to Rails application servers (e.g. Mongrel/Thin/Unicorn). While I often just stick with Apache as a web server, I've been experimenting with Nginx and so installed from source as follows:
Note: For rewrite in Nginx, the PCRE library had to be installed (See http://www.cyberciti.biz/faq/debian-ubuntu-linux-install-libpcre3-dev/):
apt-get install libpcre3 libpcre3-dev
Download the latest stable source code from http://wiki.nginx.org/NginxInstall, extract, configure and compile.
tar xzfv nginx-0.8.53.tar.gz
cd nginx-0.8.53
./configure --with-http_ssl_module
make
make install
Usergroups
Simple system administration stuff, but I like to have a staff usergroup where I put all my users. This allows for assigning permissions more easily to this group, and in turn to those users (see the following section on SSH):
addgroup <usergroupname>
adduser <username> --ingroup <usergroupname>
Lock it down!
SSH is the doorway into my server, so I like to keep it locked pretty tight. The following are a few tweaks I make to the default SSH Daemon configuration /etc/ssh/sshd_config:
Change: LoginGraceTime 120
To: LoginGraceTime 40
Change: PermitRootLogin yes
To: PermitRootLogin no
Disable password authentication in favour of RSA authentication (read more on setting this up).
Change: PasswordAuthentication yes
To: PasswordAuthentication no
Under "Authentication:" add:
MaxAuthTries 3
If a staff group exists and it has existing users, add at the end of the file:
AllowGroups staff
Finally, after saving those changes, reload the SSH daemon so they take effect (double check all your changes first, as you might lock yourself out if you made a mistake):
/etc/init.d/ssh reload
Rainer Wichmann's write-up at http://www.la-samhna.de/library/brutessh.html is well worth reading for more SSH security tips.
Monitoring
Monitoring your critical services is a must for any production system, and as far as I'm concerned you need look no further than monit. I've played around with God and while writing your monitoring configuration with Ruby is nice, I found it unstable, buggy and the memory footprint far too large, as have others. Monit is proven, stable, lightweight and configuring it is a sintch.
apt-get install monit
ImageMagick & Friends
If you do any kind of image manipulation in your Rails apps you'll inevitably need ImageMagick and RMagick (or mini_magick, ImageScience etc).
Installing these can be a real pain (see OS X!), but thankfully I found it straightforward on Debian 5, with one gotcha, I had to use rmagick v2.12.2 as later versions refused to compile.
apt-get install imagemagick
apt-get install libmagick9-dev
gem install rmagick --no-ri --no-rdoc --version=2.12.2
gem install mini_magick --no-ri --no-rdoc
Once all installed, remove libmagick9-dev and its dependencies as they are no longer needed:
apt-get remove libmagick9-dev
apt-get autoremove
Other bits and bobs
The Wordpress Google Reader module I use on this site requires the PHP Curl module, installed as follows:
apt-get install php5-curl
Wordpress also needs mod-rewrite for pretty URLs:
a2enmod rewrite
I also tend not to use sudo, instead preferring to switch user to root after I SSH into the server, so I usually remove it:
apt-get remove sudo
And don't forget to update your server's hostname:
hostname desired-host-name
Keeping things tidy
Finally, a little bit of house cleaning to remove any cruft left behind by Aptitude. Who knows, you might need the extra disc space one day!
apt-get clean
And we're done!
About Me
I'm Niall Mullally, a Web Developer based in London, UK. I specialise in the creation of modern, standards compliant websites using agile development methodologies.
- Read more about me
- View my LinkedIn profile
- Browse some of my work
What I've Been Reading
- So you want to be a video game programmer? – part 3 – Getting Started All Things Andy Gavin
- So you want to be a video game programmer? – part 1 – Why All Things Andy Gavin
- So you want to be a video game programmer? – part 2 – Specs All Things Andy Gavin
- Your Code is My Hell Virtuous Code
- Rails 3 Asset Pipeline & Google Closure igvita.com
My Twitter Feed
- The Dark Knight Rises trailer we needed. http://t.co/jEbkZCjO (via @God_Damn_Batman)
- RT @andyroberts_uk Just look how great @Jellybooks looks on different mobile devices http://t.co/jahktQuF Clever isn't it?! #ff @madebysplen
- @sandyliushengli Thanks! Getting old...
- @sandyliushengli That's how Batman was born! Your destiny awaits you Batgirl....! :-)
- @bernardonicolau Nice meeting you yesterday at @UXLondon. Enjoy the rest of your stay in London!
Open Source Projects
Stack Overflow
