Upgrading version controlled Rails gems frozen in vendor/rails

I recently had to upgrade quite an old Rails application from Rails v2.1 to v2.3.9. Previously I would do this by deleting the old rails/ directory in vendor/ and refreezing the gems. However, I have had the strange errors from Subversion when merging the upgrade branch back into trunk and it caused me a sufficient headache to find an alternative solution.

My approach now is to keep the existing vendor/rails directory and copy the new Rails files into it, doing a recursive diff of the two directories first to identify files and directories that need to be deleted as they don't exist in the new version. The process is a little bit fiddly but I've had no merge issues since adopting it, so I find it worth the extra effort.

Here's my step-by-step process:

1 - Temporarily rename vendor/rails so the latest Rails version can be frozen (all commands are assuming you in the Rails root directory):

mv vendor/rails vendor/rails_old

2 - Freeze the newer version of Rails (in this case, 2.3.9 as the app isn't quite ready for a Rails 3 upgrade yet)

rake rails:freeze:gems VERSION=2.3.9

This creates vendor/rails with version 2.3.9 of the Rails gems.

3 - Move vendor/rails_old back to vendor/rails so the subversion tracking doesn't break.

mv vendor/rails vendor/rails_new
mv vendor/rails_old vendor/rails

Do a quick 'svn status' to confirm everything is correct.

svn st vendor

It should look like this:

? vendor/rails_new

4 - Now do a diff of the two Rails directories to see what files only exist in the old version of Rails. These are the files we want to delete.

diff -qr vendor/rails vendor/rails_new | grep "Only in vendor/rails/" | grep -v -e "svn" > rails_diffs

Note the use of grep to return only files that exist in rails/. The output is also written to ./rails_diffs as we need to edit the output slightly. It will look something like this:

Only in vendor/rails/actionmailer/lib/action_mailer/vendor: tmail-1.1.0
Only in vendor/rails/actionmailer/lib/action_mailer/vendor: tmail-1.2.3
Only in vendor/rails/actionmailer/lib/action_mailer: vendor.rb
Only in vendor/rails/actionmailer/test/fixtures/first_mailer: share.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_example_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_helper_method.rhtml
Only in vendor/rails/actionmailer/test/fixtures/helper_mailer: use_mail_helper.rhtml
Only in vendor/rails/actionmailer/test/fixtures/path.with.dots/funky_path_mailer: multipart_with_template_path_with_dots.rhtml
Only in vendor/rails/actionmailer/test/fixtures/path.with.dots: multipart_with_template_path_with_dots.rhtml
Only in vendor/rails/actionmailer/test/fixtures: raw_base64_decoded_string
Only in vendor/rails/actionmailer/test/fixtures: raw_base64_encoded_string
Only in vendor/rails/actionmailer/test/fixtures/second_mailer: share.rhtml
Only in vendor/rails/actionmailer/test/fixtures/templates: signed_up.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.ignored.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.html.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.plain.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: implicitly_multipart_example.text.yaml.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up.erb
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up.rhtml
Only in vendor/rails/actionmailer/test/fixtures/test_mailer: signed_up_with_url.rhtml
Only in vendor/rails/actionpack/lib/action_controller: assertions.rb
Only in vendor/rails/actionpack/lib/action_controller/caching: sql_cache.rb
Only in vendor/rails/actionpack/lib/action_controller/cgi_ext: session.rb
Only in vendor/rails/actionpack/lib/action_controller: components.rb
Only in vendor/rails/actionpack/lib/action_controller: request_profiler.rb
Only in vendor/rails/actionpack/lib/action_controller: routing_optimisation.rb
Only in vendor/rails/actionpack/lib/action_controller/session: active_record_store.rb
Only in vendor/rails/actionpack/lib/action_controller/session: drb_server.rb
Only in vendor/rails/actionpack/lib/action_controller/session: drb_store.rb
Only in vendor/rails/actionpack/lib/action_view: compiled_templates.rb
Only in vendor/rails/actionpack/lib/action_view/helpers: javascripts
Only in vendor/rails/actionpack/lib/action_view: partial_template.rb
Only in vendor/rails/actionpack/lib/action_view: template_finder.rb
Only in vendor/rails/actionpack/lib/action_view/template_handlers: compilable.rb
Only in vendor/rails/actionpack/test: action_view_test.rb
Only in vendor/rails/actionpack/test/controller: cgi_test.rb
Only in vendor/rails/actionpack/test/controller: components_test.rb
Only in vendor/rails/actionpack/test/controller: custom_handler_test.rb
Only in vendor/rails/actionpack/test/controller: fragment_store_setting_test.rb
Only in vendor/rails/actionpack/test/controller: http_authentication_test.rb
Only in vendor/rails/actionpack/test/controller: integration_upload_test.rb
Only in vendor/rails/actionpack/test/controller: new_render_test.rb
Only in vendor/rails/actionpack/test/controller: session_fixation_test.rb
Only in vendor/rails/actionpack/test/controller: session_management_test.rb
Only in vendor/rails/actionpack/test/fixtures/test: block_content_for.erb
Only in vendor/rails/actionpack/test/fixtures/test: erb_content_for.erb
Only in vendor/rails/actionpack/test/template: deprecated_erb_variable_test.rb
Only in vendor/rails/actionpack/test/template: template_finder_test.rb
Only in vendor/rails/actionpack/test/template: template_object_test.rb
Only in vendor/rails/activerecord/lib/active_record: vendor
Only in vendor/rails/activerecord/test: aaa_create_tables_test.rb
Only in vendor/rails/activerecord/test: abstract_unit.rb
Only in vendor/rails/activerecord/test: active_schema_test_mysql.rb


5 - All the files we want to delete are in rails_diffs, so using your favourite text editor do a I find of all instances of "Only in", replacing it with "svn rm" and also find ": " (note single white space) and replace it with "/". This will modify the file to look something like:

svn rm vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.1.0
svn rm vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.2.3
svn rm vendor/rails/actionmailer/lib/action_mailer/vendor.rb
svn rm vendor/rails/actionmailer/test/fixtures/first_mailer/share.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_example_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_helper_method.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/helper_mailer/use_mail_helper.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/path.with.dots/funky_path_mailer/multipart_with_template_path_with_dots.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/path.with.dots/multipart_with_template_path_with_dots.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/raw_base64_decoded_string
svn rm vendor/rails/actionmailer/test/fixtures/raw_base64_encoded_string
svn rm vendor/rails/actionmailer/test/fixtures/second_mailer/share.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/templates/signed_up.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.ignored.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.html.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.plain.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/implicitly_multipart_example.text.yaml.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.erb
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.rhtml
svn rm vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up_with_url.rhtml
svn rm vendor/rails/actionpack/lib/action_controller/assertions.rb
svn rm vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb
svn rm vendor/rails/actionpack/lib/action_controller/cgi_ext/session.rb
svn rm vendor/rails/actionpack/lib/action_controller/components.rb
svn rm vendor/rails/actionpack/lib/action_controller/request_profiler.rb
svn rm vendor/rails/actionpack/lib/action_controller/routing_optimisation.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/active_record_store.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/drb_server.rb
svn rm vendor/rails/actionpack/lib/action_controller/session/drb_store.rb
svn rm vendor/rails/actionpack/lib/action_view/compiled_templates.rb
svn rm vendor/rails/actionpack/lib/action_view/helpers/javascripts
svn rm vendor/rails/actionpack/lib/action_view/partial_template.rb
svn rm vendor/rails/actionpack/lib/action_view/template_finder.rb
svn rm vendor/rails/actionpack/lib/action_view/template_handlers/compilable.rb
svn rm vendor/rails/actionpack/test/action_view_test.rb
svn rm vendor/rails/actionpack/test/controller/cgi_test.rb
svn rm vendor/rails/actionpack/test/controller/components_test.rb
svn rm vendor/rails/actionpack/test/controller/custom_handler_test.rb
svn rm vendor/rails/actionpack/test/controller/fragment_store_setting_test.rb
svn rm vendor/rails/actionpack/test/controller/http_authentication_test.rb
svn rm vendor/rails/actionpack/test/controller/integration_upload_test.rb
svn rm vendor/rails/actionpack/test/controller/new_render_test.rb
svn rm vendor/rails/actionpack/test/controller/session_fixation_test.rb
svn rm vendor/rails/actionpack/test/controller/session_management_test.rb
svn rm vendor/rails/actionpack/test/fixtures/test/block_content_for.erb
svn rm vendor/rails/actionpack/test/fixtures/test/erb_content_for.erb
svn rm vendor/rails/actionpack/test/template/deprecated_erb_variable_test.rb
svn rm vendor/rails/actionpack/test/template/template_finder_test.rb
svn rm vendor/rails/actionpack/test/template/template_object_test.rb
svn rm vendor/rails/activerecord/lib/active_record/vendor
svn rm vendor/rails/activerecord/test/aaa_create_tables_test.rb
svn rm vendor/rails/activerecord/test/abstract_unit.rb
svn rm vendor/rails/activerecord/test/active_schema_test_mysql.rb


This creates a list of commands to run to remove those files/directories from version control. So copy the contents of rails_diff and paste it into your command line. You'll see a bunch of output confirming Subversion has removed those files/directories.

6 - While "svn rm" will physically delete files, directories still remain, so lets tidy those up to. Go back to rails_diff and find all instances of "svn rm", replace with "rm -rf" and again copy and paste the contents of the file into the command line. Yes this will try and delete files that have already been deleted but as we're forcing the delete with the '-f' option, it won't complain.

7 - Now that all the unneeded files from the old version of Rails have been removed it's time to update vendor/rails/ to the our newer version.

cp -R vendor/rails_new/ vendor/rails/

This will modify existing files that have changed in the newer version of Rails and add any new files. Modified files are already tracked by Subversion, but we need to add the new files to version control. So in a similar way to how we diff'd the two rails directories, we'll use 'svn status' to tell us what files are new.

svn st vendor/rails | grep ? > rails_diffs

The output again goes into ./rails_diff and will look something like:

? vendor/rails/activeresource/test/debug.log
? vendor/rails/activeresource/test/http_mock_test.rb
? vendor/rails/activeresource/test/fixtures/proxy.rb
? vendor/rails/activeresource/lib/active_resource/exceptions.rb
? vendor/rails/actionmailer/test/mail_layout_test.rb
? vendor/rails/actionmailer/test/asset_host_test.rb
? vendor/rails/actionmailer/test/fixtures/auto_layout_mailer
? vendor/rails/actionmailer/test/fixtures/layouts
? vendor/rails/actionmailer/test/fixtures/asset_host_mailer
? vendor/rails/actionmailer/test/fixtures/explicit_layout_mailer
? vendor/rails/actionmailer/test/fixtures/test_mailer/body_ivar.erb
? vendor/rails/actionmailer/test/fixtures/test_mailer/signed_up.html.erb
? vendor/rails/actionmailer/lib/action_mailer/vendor/tmail.rb
? vendor/rails/actionmailer/lib/action_mailer/vendor/tmail-1.2.7
? vendor/rails/actionmailer/lib/action_mailer/vendor/text_format.rb
? vendor/rails/activesupport/lib/active_support/message_verifier.rb
? vendor/rails/activesupport/lib/active_support/memoizable.rb
? vendor/rails/activesupport/lib/active_support/secure_random.rb
? vendor/rails/activesupport/lib/active_support/message_encryptor.rb
? vendor/rails/activesupport/lib/active_support/locale
? vendor/rails/activesupport/lib/active_support/rescuable.rb
? vendor/rails/activesupport/lib/active_support/all.rb
? vendor/rails/activesupport/lib/active_support/xml_mini.rb

8 - Do a find of '?' and replace it with 'svn add'. Again, copy and paste the file contents into the command line. This will add all these files to version control. Once complete, check the svn status of vendor/rails again to confirm that everything is now tracked.

svn st vendor/rails | grep ?

If this outputs nothing then you're good to go.

9 - Your vendor/rails directory is now up to date and ready to commit to your repository.

svn commit vendor/rails -m "Upgraded vendor/rails from vX.X.X to vX.X.X"

10 - Finally a bit of tidying up:

rm -rf vendor/rails_new
rm ./rails_diffs

All done!