Summer of Rails - Ship Your First App!

Posted by kev Thu, 06 Jul 2006 23:35:00 GMT

Where do you stand on the Rails spectrum? Are you a Ruby wiz who’s released plugins, libraries, and patches to core? More likely you’re closer to the other end of the spectrum: tinkering, interested, but haven’t shipped an app.

Well, for all the tinkerers out there you’ve got some new motivation. Patrick Crowley of sd.rb is starting a project he’s calling Summer of Rails. It’s sort of a Rails Day meets Summer of Code meets “get your butt in gear and ship something already”. It’s an excuse for everyone who’s been interested in Rails but just hasn’t done it.

So, get on it! Write your app, badge it as a Summer of Code project and stop procrastinating. As new projects start coming in they’ll be getting the spotlight at the Summer of Rails site so others can bask in your new found magnificence.

Just do it!

Posted in , ,  | 1 comment

mkrf 0.1.0 Released

Posted by kev Wed, 28 Jun 2006 10:41:00 GMT

I’m pleased to announce that tonight marks the first release of mkrf, a library which generates Rakefiles to build C extensions to Ruby and is aimed at a replacement for mkmf. This is a Ruby Summer of Code project.

Though the gem doesn’t seem to have propogated yet, it should be availble through rubygems, and until then you can download the gem directly here.

The code does have RDoc included which I hope explains things well enough. Please do send questions and bug reports my way. I’ll have a project setup at advisr.org where you can submit bug reports as soon as I tweak its source to properly scope accounts, but until then email is fine.

Please do try mkrf with your extensions. Please do send feedback.

As for syntax, it can be as simple as:

 Mkrf::Generator.new('libtrivial_so.bundle')

or more complex:

 Mkrf::Generator.new('libxml_so.bundle', '*.c') do |g|
   g.include_library('socket','socket')
   g.include_header('libxml/xmlversion.h',
                    '/opt/include/libxml2',
                    '/usr/local/include/libxml2',
                    '/usr/include/libxml2')
 end

Posted in  | 3 comments

Homesteading: A Thriver's Guide

Posted by kev Sat, 24 Jun 2006 23:36:00 GMT

Tonight after Mike Clark’s talk, Nathaniel Talbott gave an excellent follow up to his RubyConf 2005 talk, ”Serving the Long Tail in 1883 and 2005” to the full conference. Read past the jump for more.

Read more...

Mike Clark - Testing Rails Apps

Posted by kev Sat, 24 Jun 2006 21:30:00 GMT

Mike Clark is a really fantastic speaker and I feel like he sold testing to a lot of people who weren’t convinced tonight. I can’t really do him justice, so you should really take any opportunity to see him. If you weren’t able to catch his talk, my notes follow after the jump.

Read more...

Posted in ,

Glen Vanderburg - Testing Migrations

Posted by kev Sat, 24 Jun 2006 20:00:00 GMT

Glen Vanderburg glv@vanderburg.org

About Me

  • Independent consultant
    • Ruby, Java, Agile, TDD, the usual suspects
    • Frequent conference speaker
  • “Generalizing specialist”
    • Addicted to learning
  • Rubyist since 2000
  • Rails since mid-2005
  • Previous experience: Java, C++, Perl, C, Tcl, REXX, Awk, sh, Pascal …

About this Talk

  • Work in progress.
  • Thinking about testing Rails migrations.
    • how
    • why
    • whether
    • when
    • how much
  • Preliminary code for testing migrations.
  • A challenge to take this farther.

Migrations Rock!

The Problem

  • Database schemas always need to evlove.
  • Most frameworks/platforms/tools ignore this fact.
  • Left to themselves, teams do crazy things…

A Common Result

  • ad hoc changes
  • Made manually in development or staging
  • Must be rediscovered (somehow) and recreated in test, and then productions

Another Common Result

  • Resistance to changing the db
  • Application much change instead
    • in unpleasent ways
  • Meaning of schema is perverted
  • Overloaded fields, severer denormalization, etc
  • Britleness!

Still Another Common Result

  • Versioned SQL script for schema
    • and one for basic test data
  • Not too bad .. schema all in one place
  • Version control systems help manage change
  • Development systems can often rebuild database from scratch
  • Time to deploy? look at differences, manually modify databse.

The Best of the Rest

  • Hand-written SQL migration scripts
  • Use database to dump schema for single-file view
  • Easy to move changes to test, production servers
  • Not too bad, except rollback is rarely considered…
    • and the SQL is usually pretty ugly.

Rails Does it Right

  • Migrations deal with all of those problems.
  • Nice (mostly) database independent code
  • Incremental migrations
  • up and down method provide for rollback

* AND YET *

Migrations suck (at least a little)

  • No testing support
  • It turns out to be hard to build testing support
    • and only partly for technical reasoning

Why?

  • Migrations and version control are an odd match
  • Active Record’s model abstraction leaks like a sieve
  • Old migrations don’t need ongoing testing
  • Sometimes testing migrations is overkill

Your Application and Version Control

  • Each version represents a point in time
    • start from scratch, check out, run your app
    • that’s the whole point!
  • Old versions are hidden
    • You can get to them, but that’s time travel

Version Control Wormhole

  • Migrations are very different.
  • The whole history of your schema in one directory
    • In one version of your directory.
  • Move the database back in time while the app stays put.
  • Causes some obvious problems.
    • Never change a migrations after you check it in”

Models or Tables?

  • Active Record lets the tables show through
  • has_and_belongs_to_many hides them (especially the join table)
  • Parameters to find (and other methods) expose them.
  • Unit tests are especially leaky places.

Fixtures: Models or Tables?

  • All information about overridden class/table mapping is in the class.
  • Then in unit test, fixtures are all about tables, so things fail.
  • How does it know about class Person?
    • See Rails ticket #1911 (flexible fixtures)

But Migrations are ALL Tables!

  • Yes they are . (Until you start thinking about testing them)
  • In migrations: add_column :events, :user_id, :integer
  • In test: assert_has_column :events, :user_id, :integer
  • Does that really help?
  • Wouldn’t this be better?
    • assert_supports_has_many :users, :events

Old Migrations Shouldn’t Break

  • Remember the version control wormhole?
  • Old migrations are history
  • If you’ve already moved beyond them (especially in production) they should never change, and should never break
  • If testing migrations is slow, don’t keep testing old migrations

Sometimes Migrations Don’t Need Testing

  • Just a few lines
  • Run promptly in development
  • Test-through-use with new application code

What’s the Point?

  • Often, the trouble of automated testing is of marginal benefit for migrations
  • There are two big exceptions
    • Data conversion
    • The down method
  • Those two things usually won’t be fully exercised until you really need them – in production

Migration Testing Setup

  • see slides (but I’ll summarize as I can)
  • Move from Category habtm Merchants to Category has many Merchants through Order
  • Lots of code examples, I’ll try to link the slides
  • Mix in MigrationTesting module (unreleased)
  • def migration_number; 2; end
  • def setup; db_setup; end

Data Conversion

  • Can sometimes be done with model instances.
  • But not in general, because you’re working across two different version of the tables underneath the models

Why Test Data Conversion?

  • Real-world data is strange.
  • Your development data probably isn’t the same.
    • (or at least, a much smaller sample size)
  • You want to test before your migration chokes on the production data.
    • Unexpected nulls can wreak havoc
  • Sample code…

Testing Up/Testing Down slides are all code examples

Why Test Reverse Migrations?

  • rake rollback really *is* the best thing since sliced bread.
    • rake deploy is merely the best thing since SSH.
  • But if your new release includes a migration, you’re counting on the reverse migrations
  • Have you ever run it?
  • Did you even write it?

Making it Easy to do the Right Thing

  • Testing reverse migrations should be trivial. (Free even)
  • The failing test provides incentive to write the reverse migration in the first place.
    • Not to mention getting it right

Testing Down

  • test_down_schema gets mixed into your test case and runs automatically
  • Detects pure schema errors for (almost) free
  • Test downward data conversion using the same techniques

What, Me Worry?

  • Maybe migratiosn don’t need to be as thoroughly tested as other code
  • But some parts do
    • Data conversion
    • Rollback code
  • Testing rollback code, in particular should be trival
    • For one thing, it prompts us to write tests

Further Work

  • Smarter “down” testing
    • Detecting errors is easy
    • Reporting them is clumsy
  • Model oriented assertions
    • Associations, acts_as, aggregation
      • Should exploit ActiveRecord reflection facilities
  • Better infrastructure
    • Automatically test new migrations, skip old ones
    • Fixture creation and management for migration testing

Posted in ,

Ezra - Deployment: Tales from the Front

Posted by kev Sat, 24 Jun 2006 18:32:00 GMT

Ezra Zygmuntowics

  • zig-mun-tuv-itch
  • Just another ruby hacker

Case Study

  • Yakima Herald-Republic Newspaper Online
  • http://yakimaherald.com
  • Launched 2005
  • Data heavy
  • Rails CMS
  • Proprietary DB called Baseview (non SQL)

Goals

  • Content management
  • Internet and Intranet
  • Automation of Classified and Obituary workflow
  • Internal credit card transactional console

Data Aggregation

  • Mysql for CMS and storage of classified and obituary data
  • XML feeds from the AP News Wire
  • RSS and Atom feeds from parent newspaper The Seattle Times

Proprietary Database

  • Problems:
    • Non-SQL BaseView Live-IQ db holds 80% of all content
    • Only accessible from poorly implemented Live-IQ Script

Solution: ruby DSL

  • Emits LiveIQ Script
  • Sends it down a socket to the LiveIQ Script interpreter
  • No more LiveIQ Script! Ruby all the way baby.

Deployment Round 1

  • Apache with FastCGI
  • Acceptable performance
  • Industry Standard
  • But…
  • has_many :fastcgi_500_errors :)

Apache/fcgi == Unstable

  • 500 errors for no apparent reason
  • zombie processes

Round 2

  • Lighttpd and mod_fastcgi for teh win!
  • Started with 4 static fcgi listeners.
  • Very good performance and stability.
  • This was when lightty didn’t run the max procs

Lighttpd

  • Very fast for static files
  • Good FastCGI support

Still needed work

  • Still doing deploys by hand
  • Still having some problems

Work in Progress

  • Switch from static fcgi procs to seperately spawned setup
  • Each fcgi on it’s own IP:PORT
  • Switched to spinner/spawner/reaper
  • Started using Capistrano

Stable.. finally

  • Exernal fcgi’s much more stable

Tuning and tweaking

  • Cache as much as you can
  • Get as much RAM as you can
  • Find hot spots and externalize them
  • Don’t guess, measure.
    • Tiny increments, measure performance.
  • httperf not ab
    • ab’s a pile” – Ezra
  • http://zedshaw.com/blog/philosophy/itanalysisrubric.html

Stats

  • 4 months start to finish
  • One developer one designer
  • 150,000+ page views daily
  • 40% page cached
  • 60% dynamic
  • Running great for 12 months now

Shared hosting setup

    • Each app gets its own port with an instance of lighty.
    • Users can restart their apps without restarting apache
    • Performance hit from too much indirection (sometimes 10-20%)
  • Decent trade offs for shared environment

Face it.. Shared hosting + Rails == Suck

  • Only as reliable as the worst app on the box.
  • Not everyone is a good neighbor
  • Constraints on resources
  • Each man for himself
  • If you can avoid it, please do

Get a VPS

  • Rails really thrives when you have your own guaranteed resources.
  • Get a host that uses Xen
    • More of a performance hit on Virtuoso Linux etc
  • Start with a minimum of 128Mb - 192+ RAM
  • http://brainspl.at/rails_stack.html

Rails uses a 3 tier architecture

  • Capistrano roles: Web (lighttpd), App: (fcgi or mongrel), DB (mysql or postgres)
  • Web: static images and assets, cached pages
  • App: rails application processes
  • DB: database daemon

As you scale..

  • Move db to it’s own machine
  • Use most powerful box for db
  • Spawn fcgi procs on db box as needed
    • 3 or 4 will do alot of traffic

Split to 3 roles

  • Typical Lighttpd Three Tier Cluser
    • Web, app and db on own boxes
  • Proven scalable setup

But things always change…

  • The future of rails deployment is mongrel

Mongrel is a good dog

  • Mongrel is a Ruby/C webserver written by my hero Zedas Shaw
  • Simple, fast and reliable.
  • Easily extensibly via gem plugins

Advantages

  • Mongrel speaks plain old HTTP
  • Can take advantage of the wealth of tools available for HTTP load-balancing and proxying.
  • Opens up many more possibilities for front ends.

Front End

  • Pound, Pen, Balance, Haproxy or even hardware load balancers
  • Lighttpd (although mod_proxy needs to be fixed)
  • Apache 2.2 mod_proxy_balancer
    • I’m compiling Apache with only mod_proxy, mod_proxy_balancer, mod_deflate which works great
    • Gives real time status report of backends and lets you restart things

Apache 2.2

  • Apache is an industry standard and is working well with mongrel

Mongrel Cluster

  • Written by Bradley Taylor
  • Allows for easy management of clusters of mongrel processes
  • Very easy to Script with Capistrano

$sudo gem install railsmachine

  • Also by Bradley Taylor
  • Capistrano task library
  • Creates svn repo and imports your app
  • Configures Apache virtual hosts with support for mod_proxy_balancer, ssl, mod_deflate
  • Confugres mongrel_cluseter
  • Creates MySQL db
  • Creates symlinks for items in /public for use with file_column

Clustered Rails Deployment with Engine Yard

  • My pet project
  • Easy push button deployment through a web interface
  • Each application runs on two Xen VPSs each on a separate physical box
  • Redundant load balanced for fallover
  • Add and remove cluster nodes at will
  • Configure ‘packages’ that consist of an svn revision or tag of your app and frozen, versioned gem dependencies.
  • Drag & drop to deploy your packaged app from Stage to QA to Production

BackgrounDRb

  • Solution for forking, shelling for a binary etc
  • Distributed ruby server for managing long running background tasks
  • Available as a rails plugin
  • MiddleMan object for use in rails to control tasks
  • Hooks for ajax progress bars or status updates while task runs

Great code example slides here. Look for slides on http://brainspl.at

One last plug

  • Rails Deployment Book
  • Pragmatic Programmers
  • Beta “Real Soon Now”

Q/A

  • Is the ruby interface you created for IQ Script proprietary?
    • Unfortunately yes, the newspaper owns it, but they’re thinking of releasing it. Watch my blog.
  • Why did you choose the cocoa interface over Rails for the internal interface?
    • It made it alot easier to deal with because of file constraints. Cocoa guis are really nice and it’s easy to glob up a bunch of files. It was fun too :). It could have been a Rails interface. Handling a couple hundred files at once is cumbersome through the web.
    • mod_fcgid?
      • Some people have had better luck with it than regular fcgi. fcgid is a daemon so it’s better at killing zombies, but it still gets the 500 errors. Not as much, but after wrestling with apache for a long time I moved off it.
    • How do you look at Xen in comparison to other VMs?
      • I like Xen because it’s in the kernel. VMWare is another beast. It’s more for quick testing on my workstation. I haven’t use a professional grade VMWare though, but I really like Xen. One of the great things is that the host can’t oversell the resources. With Virtuoso you can oversell which gives you burst limits. With Xen there’s no way for your host to oversell the box.
    • What kind of hardware do you prefer?
      • We’ve built whitebox boxes with opterons (dual, 4 gig). We were going to go to Rackspace but we got our own cage in California.
    • Do you have more information on the 500 errors or why they happen?
      • There’s timeouts. I haven’t dug in the C code, but fcgi’s management process is flakey. There has been some work on a mod_proxy_fastcgi for the newer apache but nothing is released yet. I look forward to seeing how it works. The general feeling for me is that fcgi isn’t very stable.
    • In Engine Yard, how do you deal with migrations and error handling?
      • We’ll have some servers dedicated to staging. An identical environment to yours before you push to production. You’ll be able to run your rake tasks and such through the db.
    • Do you have extra suggestions for somebody who starts a new project regularly (we have dozens of apps). Some are small (80%), some are big (20%). We don’t want apps to destroy each other.
      • I’d think that for small apps it’d be nice to consolidate into one app (if they’re all pretty much the same), but mongrel is definitely the way to go. 1 per app probably. One nice solution would be to get a dedicated box and to run one xen instance with Apache with virtual hosts and then other xen instances with each app.
    • How should you choose hardware or software load balancers?
      • The hardware load balancers go between the web servers. The web servers (apache) load balance to app servers (mongrel).
    • Engine Yard, anything on pricing?
      • I can’t say yet. It’ll be affordable, but we’re not looking to compete with shared hosting.
    • I work on pogo.com which is in the top 250 on Alexa ranking. Is Rails ready for a top 100?
      • I don’t see any reason why not. It’s typical LAMP share nothing, so the issues is getting your architecture straight, not Rails.

Posted in

Stefen Kaes - Optimizing Rails

Posted by kev Fri, 23 Jun 2006 22:08:00 GMT

Stefen went very very fast during his presentation, so I’ve missed bits and pieces. I’ll link his slides if I can (though they may not be available except for the $50 video). Sorry about that.

Performance Tuning

  • Trying to improve performance without measuring is foolish
  • In favor of optimization at design time

Performance Parameters

  • Latency
    • How fast can you answer a request?
  • Throughput
    • How many requests can you process per second?
  • Utilization
    • Are your servers idle most of the time?
  • Cost efficiency
    • Performance per unit cost
  • Compute mean, min, max, standard dev (if applicable). Standard deviation will tell you how reliable your data is.

Benchmarking Tools

  • Rails log files (debug level >= Logger::DEBUG)
  • Rails Analyzer Tools (requires logging to syslog)
  • Rails benchmarker script (script/benchmarker)
  • Tools provided by DB vendor
  • Apache Bench (ab or ab2)
  • httperf
  • railsbench
    • downloadable from http://rubyforge.org/projects/railsbench

railsbench

  • Measures raw performance of Rails request processing configured through:
    • benchmark definitions
      • $RAILS_ROOT/config/benchmarks.yml
      • defines which urls you want to visit in yaml
    • benchmark class configuration
      • $RAILS_ROOT/config/benchmarks.rb
      • creates a benchmarking instance with an ActiveRecordStore
      • Can also define user locking etc.
    • stores benchmark data in $RAILS_PERF_DATA
    • indexed by date and benchmark time
    • uses additional Rails environment benchmarking
  • Usage
    • perf_run 100 "-bm-welcome options" [data file]
      • Run 100 iterations of benchnmark with given options, print data
    • perf_diff 100 "-bm=all opts" "opts1" "opts2" [file1] [file2]

railsbench options

  • -log[=level]
    • turn on logging (defaults to no logging). optionally oveerride log level.
  • -nocache
    • turn off rails caching
  • -path
  • -svlPV
    • run test using Ruby Performance Validator
  • patched_gc
    • use patched GC Ruby Profiling Tools
  • Ruby Profiler
  • Zen Profiler
  • rubyprof
  • Rails profiler script
  • Ruby Performance Validator (commercial, Windows only)
  • All but the last are pretty much useless for Rails performance work.
  • railsbench has builtin support for RPVL:
    • run_urls 100 -svlPV -bm=welcome ...
  • will start RPVL and run the named benchmark with given options

Please send an email to the RPV guys if you think it should have UNIX support

Top Rails Performance Problems

  • Depends on who you ask, but these are my favorites:
    • slow helper methods
    • complicated routes
    • associations
    • retrieving too much from DB
    • slow session storage
  • Judging from my experience, DB performance is usually not a bottleneck.
  • Instantiation ActiveRecord objects is more expensive

Available Session Containers

  • In Memory
    • Fastest but you lose all sessions on server crash/restart. Restricted to 1 app. Doesn’t scale.
  • File System.
    • Easy setup, one file for each session. Scales by using NFS or NAS (beware 10k active sessions!). Slower than
  • Database/ActiveRecordStore
    • Easy setup (comes with Rails distribution). Much slower than
  • Database/SQLSessionStore
    • Uses ARStore
    • More info at http://railsexpress.de/blog/articles/2005/12/19/roll-your-own-sql-session-store
  • memcached
    • Slighly faster than SQLSessionStore. Presumably scales best. Very tunable. Automatic session cleaning. Harder to obtain statistics. setup
  • DrbStore
    • Can be used on platforms where memcached is not available.

Cachable Elements

  • Pages
    • Fastes. Complete pages are stored on the file system. Web server bypasses app for rendering. Scales through NFS or NAS. Problematic if app requires login.
  • Actions
    • Second fastest. Caches the result of invoking actions on controllers. User login id can be used as part of the storage key.
  • Fragments
    • Very useful for caching small fragments (hence the name) of HTML produced during request processing. Can be made user aware.
  • Action caching is just a special case of fragment caching.
  • Several storage containers are available for fragment caching.

Storage Options for Fragment Caching

  • In Memory
    • Very very fast. If your app is running fast enough with 1 app server process, go for it!
  • File System
    • Reasonably fast.
  • DrbStore
  • memcached

ActionController Issues

  • Components
    • I suggest to avoid components. I haven’t found any good use for them, yet.
    • Each embedded component will be handled using a fresh request cycle.
    • Can always be replace by helper methods and partials.
  • Filters
    • If you are using components, make sure you don’t rerun your filters for every request.

ActionView Issues

  • Instance Variables
    • For each request, one controller instance and one view instance will be instantiated.
    • Instance vars creatd during controller processing will be transfered to view instance
    • So: avoid creating instance vars you don’t need. (PARAPHRASE, NEED TO FIND SLIDES)

Slow Helper Methods

  • pluralize(n, 'post')
    • Creates a new inflector instance, and try to derive the correct plural. This is expensive.
    • Do pluralize(n, 'post', MISSING_ARG_NEED_TO_FIND_SLIDES) instead
  • link_to and url_for
    • Much more efficient to construct your own urls, but you only need to do it on pages with large numbers of links.

ActiveRecord Issues

  • You can prefetch associated objects using :include
    • Article.find(:all, :include => :author)
  • Use piggy backing for has_one or belongs_to relations.
    • piggy.back :author_name, :from => :author, :attributes => [:name] article = Article.find(:all, :piggy => :author) puts article.author.name

Caching Column Formatting

  • Computationally expensive transformation on AR fields can be cached (in the DB, using memcached, a DRb process)
  • Example: textilize
    • I’ve analyzed an application, where 30% cpu was saved by storing the textilized value Ruby’s Interpreter is Slow
  • no byte code, no JIT, interprets ASTs directly
  • doesn’t perform any code optimization at compiler time:
    • method inlining

Complexity of Ruby Language Elements

  • Local Var acfcess: O(1)
  • Instance Var access: expected O(1)
  • Method Call: expected O(1)
    • hash access to determine literal value {"f" => :f}
    • method search
  • Recommendation:
    • don’t add method abstractions needlessly
    • use attr_accessors as external interfaces only
    • use local variables to short circuit repeated hash access
    • Avoid repeated hash access

Caching Data in Instance Variables/Class variables

  • see slides for example

Coding Variable Caching Efficiently

  • see slides for example

Defining Constants vs. Inlining

  • see slides for example

Local Variables are Cheap

  • see slides for example

Be Careful With Regards to Logging

ObjectSpace.each_object

  • see slides for example

Ruby’s Memory Management

  • Designed for batch scripts, no long running server apps
  • tries to minimize memory usage
  • simple mark and sweep algorithm
  • uses malloc to manage contiguous blocks of Ruby objects
  • complex datastructures
    • only references to C structs are stored on Ruby heap
    • comprises strings, arrays, hashes, local variables maps, scopes etc
  • eases writing C extensions
  • Current C interface makes it hard to implement generational GC

Why Ruby GC is a problem for Rails

  • ASTs are stored on the Ruby heap and will be processed on each collection
    • usually the biggest part of non garbage for Rails apps
  • Sweep phase depends on size of heap, not size of non garbage
    • can’t increase the heap size above certain limits
  • More heap gets added, if
    • size of freelist after collection, < FREE_MIN a constant defined in gc.c as 4096
    • 200,000 heap slots are a good lower bound for live data for typical Rails applications

Improving GC Performance

  • Control GC from the Rails dispatcher:
    • RailsFCGIHandler.process! nil, 50
      • Will disable Ruby GC and call GC.start after 50 requests have been processed

Patching Ruby’s Garbage Collector

  • Download latest railsbench package. Patch Ruby using rile rubygc.patch, recompile and reinstall binaries and docs.
  • Tune GC using environment variables
  • RUBY_HEAP_MIN_SLOTS
  • RUBY_HEAP_FREE_MIN
  • RUBY_GC_MALLOC_LIMIT
  • Rec values in slides (sorry)

Compile Time Template Optimization

  • Many helper calls in Erb templates can be evaluated at template compile time.
  • <%= end_form tag %> ==> </form>
  • It’s a complete waste to do it over and over again on a per request basis.
  • For some calls, we know what the output should be like, even if we don’t have all arguments available.
  • see slides

Rails Template Optimizer

  • Uses Ryan Davis’ ParseTree package and ruby2ruby class
  • Retrieves AST of ActionView render method after initial compilation
  • Transforms AST to simplify AST
  • Optimizes AST into optimized render method

Optimizer Customization and Restrictions

  • see slides

Capistrano - Mike Clark

Posted by kev Fri, 23 Jun 2006 16:40:00 GMT

A big thanks to Jamis Buck who wrote Capistrano!

  • Is deploying new releases of your app as effortless as breathing?
  • Or is deployment choking the life out of your software?
  • Capistrano (formerly Switchtower)
    • noun - Software utility and framework that automates the deployment of web applications
    • verb - to replace mundane deployment tasks with pure joy: “Stop wasting your time, just cap it already!”
    • Capistrano automates mundane and annoyingly tedious deployment tasks.
  • It scales!
  • Getting your killer app deployed into production on launch day..
    • Should be a few minutes of prep
    • Fixing your first bug, and redeploying before anyone notices
      • A few minutes of panic
    • Deploying new releases to a cluster of machines, whenever you want, at the push of a button
      • Priceless!
  • Before we press on…
    • We’re assuming you’ve already made the tough deployment decisions…
    • Demo Assumptions
  • Lighttpd, OSX, FastCGI, Subversion, One machine, Depot
    • How do I install it?
  • gem install capistrano
  • Capify! cap --apply-to /path/to/my/app MyAppName
  • Adds rake and recipe files
    • Recipe
  • Application Name
    • set :application, "depot"
  • Repository
    • set :repository, "http://svn.yourhost.com/#{application}/trunk"
  • Roles
    • Used to define what roles which servers are playing.
    • rule :app, "app01.example.com", "app02.example.com"
    • rule :web, "www01.example.com", "www02.example.com"
    • rule :db, "db01.example.com", :primary => true
    • rule :db, "db02.example.com"
    • web is for the webserver like lightty, :app is for handlers like fastcgi, db is for sql servers like mysql
  • Deployment Root
    • Where the deployed code goes
    • set :deploy_to, "/Library/Rails/#{application}"
  • Need a deploy targer? Run once to create the deploy directories
    • cap setup
    • Demo
  • Used cap setup which created a skeleton directory structure.
    • Would have worked on a dozen servers identically if specified
  • Used cap update which checked out a new copy of the app on the server
    • Releases directory now has timestamped subdirectory with the release
    • current is a symlink to that release directory
  • Here’s where it gets fun :)
  • We update the code and run cap deploy and it goes off, checks out the new code and reloads the server.
  • We regret this. cap rollback, it flips the symlink back to the last version and reloads the server.
    • Need some downtime?
  • Disables web acces: puts up maintenance page
    • cap disable_web: puts up a maintenance screen and takes down your application
    • cab enable_web to bring it back up.
    • You need this to be truly effortless when there’s a problem
  • What’s the diff?
    • What’s deployed vs what’s in repo
    • cap diff_from_last_deploy
    • It gets even better…
  • Customizing and Extending (you know you want to)
  • Make a task

    • Similar to make or rake
    • Task can define roles to use
    • Can use run to run commands on remote server
    • Can use variables inside tasks
    • Can use multiple roles (app, web, and db)
    • Chaining tasks easily: task :status do which_ruby current_revision uptime end
    • Channels and Streams
      • tail the logfile
      • Get a channel back which can be watched
    • Code run before or after a task

      • before_deploy
      • after_update_code
    • Multiple Configurations

      • It’s just ruby!
      • Can do all sorts of conditionals
      • If we’re in the production environment… else setup the dev environment
    • But wait, there’s more
  • Task Libraries
    • Can reuse tasks across projects
    • capistrano-ext - great example of how to write extension libraries
      • Great example of how to write task libraries. Has watch_load task.
    • But this isn’t just a Rails or Ruby thing…
  • It’s great at shuttling files along. Great for other systems as well.
  • Remote Command and Control
  • James Duncan Davidson has used it for numbers of servers in India as well as the US
    • Capistrano just assumes…
  • Remote servers talk POSIX
  • Same deploy directory structure and password on each machine
    • Can use public keys (and does by default)
  • Web app uses FastCGI with Apache or LightTPD
    • But can rewrite the restart task if you’d like for mongrel etc.
    • As the number of machines and processes in your environment increases…
  • you’re still typing exactly one command
    • Can you deploy and roll back releases this easily and consistently on your project?

Q&A

Mike took questions from the crowd.

  • How do you deal with permissions with multiple users?
    • We use after hooks to deal with chown etc.
    • Configurable option to use sudo
  • In these examples did your app server checkout from subversion?
    • It did a checkout on my machine. You can configure it to do an export instead. The servers do need access to version control.
  • If you’re always checking out the same code to the different servers.. is there ever a time you’re just checking out different files to servers with different roles?
    • It needs all the code for migrations generally, but you’d override one of the tasks for a specific role.
  • Sometimes I like to deploy alot but some are pretty big (30 or 40 megs of media files). Is there some way to not checkout each time?
    • I’ve seen grassroots stuff to checkout once and rsync, but you can override the update task to deal with that.
    • We’ve done that (someone in the crowd) that solves the problem.
  • Who do I talk to about getting my tasks to capistrano into the mainline?
    • You don’t. Publish task libraries and let the community get in if they’d like.
  • Can I make capistrano prompt the user for which tag in subversion they’d like to use?
    • Sure. You can specify the revision number and write ruby to prompt.
  • How did you feel about Dave Thomas’s suggestions for changes to capistrano in his keynote?
    • This isn’t the end goal. I think there will be a natural progression and “Capistrano II will” come out eventually.
  • How does capistrano deal with database migrations?
    • There is a task called deploy_with_migrations which deals with it. This seems a bit scary, so I’d probably disable the web first, but the support is certainly there.
  • If you have a cluser of 10 machines do they go down in sequence?
    • It’s done in parallel talking to each machine as best as possible.
  • We’re in a Java shop, how can we use capistrano to tail logs and things like that?
    • Capistrano could definitely move files (WAR files often) around for you. If you find yourself doing something manually in that setting, write your own tasks. You can always just write your own specific tasks and don’t have to use cap deploy.
  • When will there be a new version of “Project Automation”?
    • Life gets in the way, so there’s nothing in the works.

Posted in

ARTS and Test::Unit get a bit cosier

Posted by kev Fri, 23 Jun 2006 14:50:00 GMT

I was able to find Nathaniel Talbot to talk about why installing ARTS makes tests run during migrations and other rake tasks and as a consequence we’ve squashed the bug.

Feel free to update your ARTS plugin. Note that the latest copy does have regex matching for content.

Posted in , ,

Dave Thomas Keynote

Posted by kev Fri, 23 Jun 2006 14:37:00 GMT

What a year

  • 21 million hits on Google
  • Sidebar (how google makes their money) Advertising
    • Hosting
    • Training
    • Consulting
    • Custom project work
    • 65% of the No Fluf speakers are doing Rails work
    • Commercial IDEs for Rails
    • Most significant, ALTERNATIVES to Ruby on Rails
  • Rubyforge Downloads
    • Rails - 536,835 downloads
  • Ratings
    • Google Trends
      • Constantly increasing graph
      • In comparison to Websphere which is declining and evening out
      • Similar situation with jboss
      • We’ve passed Tapestry
      • We passed the “spring framework” a long time ago
      • We’ve passed zend
  • Thanks core!
  • Thanks Matz!
  • Everything is Perfect! BUT
    • Some work still needs doing
    • Hilbert, a mathmetician, gave a keynote analyzing the 23 most important unsolved problems in mathematics, setting the stage for the next century

23 Unsolved Problems Ran for about 7.5 hours, so…

3 Problems

  • PDI - Please Do Investigate
    • A bit rude
    • BUT, they’re right. There’s only 12 of them.
    • We’re all programmers. Try to implement it. It’s open.
    • The responsibility to make Rails the way we want is up to us.

Data Integration

  • Better use of schema - use schema constraints
    • Validation based on the schema (staying DRY)
    • Work with database foreign keys
      • It will help with enterprise integration
      • Make it easy to define in migrations
      • Add belongs_to if FK detected
      • Generally make folks feel we care
  • Primary keys
    • Better support for non-integer keys
      • Particularly in migrations
  • Add support for composite primary keys
  • Support distributed transactions
  • Standardized attribute-based finders
  • Non-database models (including JMS/MQ)

Real-world CRUD

  • The Irony - Scaffolding brings people to Rails BUT it’s the worst of Web 1.0
  • We need something that really handles the database
  • Support for table relationships
  • Configurability
  • In browser validation
  • AJAX!
  • Cross application skinning
  • What We’re Doing
    • Bring the simplicity of Active Record to Views and Controllers

Deployment

  • What do you do on the server?
    • Lightty, Mongrel, Apache, handling zombies?
  • Capistrano
    • Best deployment system there is
    • Lets make it better
    • Push model
      • Application contains all of the configuration information
      • What, where, how and when of deployment
      • Great when you own the entire development
      • BUT, in the real world responsibilities are split
      • Developers know what to deploy
      • Server administrators know where when and how
  • Capistrano II
    • Cooperative development
    • Decouple application requirements and server environments
    • Application Config
      • Where to find files
      • Where to put files
      • Pre and post development hooks
      • Library and environment prereqs
    • Server Config
      • Nominate server roles
      • Where files go on the server
      • User names/permissions/security
      • Including database passwords etc
    • Deployment
      • Set up server once
      • Set up individual applications as needed
      • cap --deploy-on cap://server.com
      • Sends application config to server
        • Server checks dependencies, etc
      • Stages application to server(s)
      • Installs (or not)
      • Then..
      • ISPs set up standard environment
      • Developers and users deploy:
      • svn co http://typosphere.org/typo cd typo cap --deploy-on cap://my.isp.com/dave
      • No more cron jobs for dead processes
      • Packaging
      • One step further…
        • Deploy from GEMS
        • Rails equivalent of WAR files
        • Deploy from GEM using:
        • gem deploy name --on cap://my.isp.com
        • gem update name --on cap://my.isp.com

Why?

  • We feel sorry for them
  • I couldn’t do Java development anymore
    • Rails and Ruby have corrupted me
  • It’s really hard to go back after Rails
  • I’d like the Rails community to embrace these people
    • Meet them halfway
    • Support what they need for the transition
  • All developers deserve to be happy

Posted in

Older posts: 1 ... 5 6 7 8 9 ... 17