Data migrations in Ruby on Rails

TL; DR Please move the data migration code into Rake tasks or use full-fledged schema-style gems. Cover this logic with tests.

I am working as a backend developer at FunBox. In a number of projects, we are writing a Ruby On Rails backend. We strive to build adequate development processes, therefore, when faced with a problem, we try to comprehend it and develop methodological recommendations. So it happened with the problem of data migration. Once I did data migration in a separate Rake task covered with tests, and the team had a question: "Why not in schema migration?" I asked the developers in the internal chat and, much to my surprise, opinions were divided. It became clear that the question was ambiguous and worthy of thoughtful analysis and an article. The maximum program in terms of goals for an article will be fulfilled for me when someone cites a link to this text in the code review in response to the question why a particular data migration was taken out or, on the contrary, not taken out from the schema migration.

Lyrical digression

, . . « . ». « ».

IT . Ruby, - . , , . , : , , , , , , .

, , . , , , , .

Rails , , DDL-. , . , Rails omakase- . , .

, , . , . , , . .

. , . (, ) , .

, . . -, , .

deadlocks .

, , , Zero Downtime Migrations Strong Migrations.

— DSL (Domain Specific Language) Ruby DDL- SQL . DSL, , . , .

DSL, , SRP. . , , …

( , )

Ruby On Rails Data Migration , . , . Rails-, . .

SQL ORM ActiveRecord.

:

1. . .
2. , .
3. callbacks , .

«» . , .

«» Rails ( 4.2):

    # db/migrate/20100513121110_add_flag_to_product.rb

    class AddFlagToProduct < ActiveRecord::Migration
      class Product < ActiveRecord::Base
      end

      def change
        add_column :products, :flag, :boolean
        Product.reset_column_information
        Product.all.each do |product|
          product.update_attributes!(:flag => false)
        end
      end
    end

.

, each find_each c batch-.

, , 4.2 .

SQL

, , SQL, :

1. , . , , , (SQL), .
2. JOIN-, , .
3. , deadlock.

,

, .

, nullable- .

, .

, :

    UPDATE table SET field = 'f' WHERE field IS NULL

:

    class ClientDemandsMakeApprovedNullable < ActiveRecord::Migration
      def up
        change_column_null :client_demands, :approved, true
        change_column_default :client_demands, :approved, nil
      end

      def down
        execute("UPDATE client_demands SET approved = 'f' WHERE approved IS NULL")
        change_column_null :client_demands, :approved, false
        change_column_default :client_demands, :approved, false
      end
    end

, . , , . Dan Mayer Managing DB Schema & Data Changes Modifying Large Tables.

. «», . . , , , .

, .

, , .

, . . REPL .

, :

1. ;
2. ;
3. .

. , . . .

, , , , .

Rake-

, — Rake-. . -.

Rake- . , . . . — .

, , Rake, Thoughtbot:

    # lib/tasks/temporary/users.rake
    namespace :users do
      desc "Actualize achievements counter cache"
      task actualize_achievements_counter_cache: :environment do
        # C (ActiveRelation)   
        users = User.with_achievements
        #    
        puts "Going to update #{users.count} users"
        # ,   ,  
        #    .    
        ActiveRecord::Base.transaction do
          # Batch-   find_each
          users.find_each do |user|        
            #     
            user.actualize_achievements_counter_cache!
            #  
            print "."
          end
        end

        puts "Done!"
      end
    end

each find_each, . memory bloats. Akshay Mohite.

. , Rake- .

, . . , , . .

Mark Qualie up, . «» . :

    class AddLastSmiledAtColumnToUsers < ActiveRecord::Migration[5.1]
      def change
        add_column :users, :last_smiled_at, :datetime
        add_index :users, :last_smiled_at
      end

      class Data
        def up
          User.all.find_in_batches(batch_size: 250).each do |group|
            ActiveRecord::Base.transaction do
              group.each do |user|
                user.last_smiled_at = user.smiles.last.created_at
                user.save if user.changed?
              end
            end
          end
        end
      end
    end

:

    Dir.glob("#{Rails.root}/db/migrate/*.rb").each { |file| require file }
    AddLastSmiledAtColumnToUsers::Data.new.up

Job, .

, - , .

, , .

, . , .

data-migrate (> 670), , Readme. Rails 5+.

, Rails 4+:

• rails-data-migrations (> 93 )

  
  
  

• nonschema_migrations (> 53 )

  
  
  

. .

, Rake-. . , .

db/data, db/migrate c :

rails g data_migration add_this_to_that

:

rake data:migrate
rake db:migrate:with_data
rake db:rollback:with_data
rake db:migrate:status:with_data

, .

.

— , :

1. — ;
2. Zero Downtime Deployment — , ;
3. Test First — ;
4. — ;
5. — , ;
6. — , , .

, Rake- — .

, . .

, — , . , .

• Rails Guides.
• Thoughtbot. Data Migrations in Rails.
• AtomicObject. Testing Data Migrations in Rails.
• Marcqualie. Rails Data Migrations.
• Ombulabs. Three Useful Data Migration Patterns for Rails.
• Strong Migrations.
• Zero Downtime Migrations.
• Dan Mayer. Managing DB Schema & Data Change.
• Akshay Mohite find_each each.
• . « . ». 1, 8 . , , .

UPD 2020-08-06: Extrapolator « ».

- , Test First .

.

UPD 2020-08-07: . , . . , .

UPD 2020-08-08: , . , . , , . « », , . . , .