Glen Vanderburg - Testing Migrations

kev — Sat, 24 Jun 2006 16:00:00 -0400

Glen Vanderburg glv@vanderburg.org

About Me

Independent consultant
- Ruby, Java, Agile, TDD, the usual suspects
- Frequent conference speaker
“Generalizing specialist”
- Addicted to learning
Rubyist since 2000
Rails since mid-2005
Previous experience: Java, C++, Perl, C, Tcl, REXX, Awk, sh, Pascal …

About this Talk

Work in progress.
Thinking about testing Rails migrations.
- how
- why
- whether
- when
- how much
Preliminary code for testing migrations.
A challenge to take this farther.

Migrations Rock!

The Problem

Database schemas always need to evlove.
Most frameworks/platforms/tools ignore this fact.
Left to themselves, teams do crazy things…

A Common Result

ad hoc changes
Made manually in development or staging
Must be rediscovered (somehow) and recreated in test, and then productions

Another Common Result

Resistance to changing the db
Application much change instead
- in unpleasent ways
Meaning of schema is perverted
Overloaded fields, severer denormalization, etc
Britleness!

Still Another Common Result

Versioned SQL script for schema
- and one for basic test data
Not too bad .. schema all in one place
Version control systems help manage change
Development systems can often rebuild database from scratch
Time to deploy? look at differences, manually modify databse.

The Best of the Rest

Hand-written SQL migration scripts
Use database to dump schema for single-file view
Easy to move changes to test, production servers
Not too bad, except rollback is rarely considered…
- and the SQL is usually pretty ugly.

Rails Does it Right

Migrations deal with all of those problems.
Nice (mostly) database independent code
Incremental migrations
up and down method provide for rollback

* AND YET *

Migrations suck (at least a little)

No testing support
It turns out to be hard to build testing support
- and only partly for technical reasoning

Why?

Migrations and version control are an odd match
Active Record’s model abstraction leaks like a sieve
Old migrations don’t need ongoing testing
Sometimes testing migrations is overkill

Your Application and Version Control

Each version represents a point in time
- start from scratch, check out, run your app
- that’s the whole point!
Old versions are hidden
- You can get to them, but that’s time travel

Version Control Wormhole

Migrations are very different.
The whole history of your schema in one directory
- In one version of your directory.
Move the database back in time while the app stays put.
Causes some obvious problems.
- “Never change a migrations after you check it in”

Models or Tables?

Active Record lets the tables show through
has_and_belongs_to_many hides them (especially the join table)
Parameters to find (and other methods) expose them.
Unit tests are especially leaky places.

Fixtures: Models or Tables?

All information about overridden class/table mapping is in the class.
Then in unit test, fixtures are all about tables, so things fail.
How does it know about class Person?
- See Rails ticket #1911 (flexible fixtures)

But Migrations are ALL Tables!

Yes they are . (Until you start thinking about testing them)
In migrations: add_column :events, :user_id, :integer
In test: assert_has_column :events, :user_id, :integer
Does that really help?
Wouldn’t this be better?
- assert_supports_has_many :users, :events

Old Migrations Shouldn’t Break

Remember the version control wormhole?
Old migrations are history
If you’ve already moved beyond them (especially in production) they should never change, and should never break
If testing migrations is slow, don’t keep testing old migrations

Sometimes Migrations Don’t Need Testing

Just a few lines
Run promptly in development
Test-through-use with new application code

What’s the Point?

Often, the trouble of automated testing is of marginal benefit for migrations
There are two big exceptions
- Data conversion
- The down method
Those two things usually won’t be fully exercised until you really need them – in production

Migration Testing Setup

see slides (but I’ll summarize as I can)
Move from Category habtm Merchants to Category has many Merchants through Order
Lots of code examples, I’ll try to link the slides
Mix in MigrationTesting module (unreleased)
def migration_number; 2; end
def setup; db_setup; end

Data Conversion

Can sometimes be done with model instances.
But not in general, because you’re working across two different version of the tables underneath the models

Why Test Data Conversion?

Real-world data is strange.
Your development data probably isn’t the same.
- (or at least, a much smaller sample size)
You want to test before your migration chokes on the production data.
- Unexpected nulls can wreak havoc
Sample code…

Testing Up/Testing Down slides are all code examples

Why Test Reverse Migrations?

rake rollback really *is* the best thing since sliced bread.
- rake deploy is merely the best thing since SSH.
But if your new release includes a migration, you’re counting on the reverse migrations
Have you ever run it?
Did you even write it?

Making it Easy to do the Right Thing

Testing reverse migrations should be trivial. (Free even)
The failing test provides incentive to write the reverse migration in the first place.
- Not to mention getting it right

Testing Down

test_down_schema gets mixed into your test case and runs automatically
Detects pure schema errors for (almost) free
Test downward data conversion using the same techniques

What, Me Worry?

Maybe migratiosn don’t need to be as thoroughly tested as other code
But some parts do
- Data conversion
- Rollback code
Testing rollback code, in particular should be trival
- For one thing, it prompts us to write tests

Further Work

Smarter “down” testing
- Detecting errors is easy
- Reporting them is clumsy
Model oriented assertions
- Associations, acts_as, aggregation
  - Should exploit ActiveRecord reflection facilities
Better infrastructure
- Automatically test new migrations, skip old ones
- Fixture creation and management for migration testing

Gluttonous: Glen Vanderburg - Testing Migrations

Glen Vanderburg - Testing Migrations