Skip to content

MongoDB->Postgres Migration (Part 1 of 2): Migrate team data from Mongoid to ActiveRecord #227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 29 commits into from
Nov 10, 2014
Merged

MongoDB->Postgres Migration (Part 1 of 2): Migrate team data from Mongoid to ActiveRecord #227

merged 29 commits into from
Nov 10, 2014

Conversation

just3ws
Copy link
Contributor

@just3ws just3ws commented Oct 15, 2014

DO NOT MERGE. UNDEAD CODE INSIDE.

Part 2 contains the code conversion that will be applied after we successfully migrate the data into the new models.

Well, it's been something we've been talking about for a while. Getting the Team implementation of of Mongoid/MongoDB and over into ActiveRecord/Postgresql. Now we're finally going to push this last mile and gitter dun.

  • Data migration
    • Create ActiveRecord models to replace the Mongoid models (@seuros)
    • Create a data migration utility to copy the data from MongoDB-backed models into Postgresql-backed models. (@seuros)
    • Create a verification utility to verify that the data is successfully migrated. (@just3ws)
    • Confirm that all the data is successfully migrated using production data. (@just3ws)
    • Deploy the new models to production (@just3ws)
    • Run the migration and verification tools in production (@just3ws)

@just3ws just3ws mentioned this pull request Oct 15, 2014
Merged
@just3ws
Copy link
Contributor Author

just3ws commented Oct 15, 2014

Update on the Migration status these are some of the data migration issues that need to be taken care of. The mostly boil down to...

  • Migrating the avatars
  • Rounding errors on mean, score, and total fields. (We need to change to a higher precision Decimal column instead of a Float.
  • Slug generation is incorrect and inserting - in unexpected places (first character?)
  • Locations count is 1 or more in Postgres but 0 in MongoDB
  • pending_join_requests references to a Array of type String in Postgres but is Array of type Integer in MongoDB
  • Some address and state_code are incorrect (see 1217)
PG Attribute Mongo Error
1000 avatar 4f27196f973bf00004000c51 /assets/team-avatar.png != /uploads/team/avatar/4f27196f973bf00004000c51/logo.png
1000 mean 4f27196f973bf00004000c51 4.734375 != 4.734375000000003
1000 members 4f27196f973bf00004000c51 16 != 17
1000 score 4f27196f973bf00004000c51 154.981484187932 != 154.9814841879325
1000 total 4f27196f973bf00004000c51 75.75 != 75.75000000000004
10000 mean 51967ed2a3fb760005000005 3.08333333333334 != 3.0833333333333357
10000 score 51967ed2a3fb760005000005 134.664452494643 != 134.66445249464317
10003 mean 5197ed0fb12dff000b000003 8.68086419753086 != 8.680864197530864
10003 pending_join_requests 5197ed0fb12dff000b000003 ["72261"] != [72261]
10003 score 5197ed0fb12dff000b000003 201.663889828456 != 201.66388982845638
10003 total 5197ed0fb12dff000b000003 234.383333333333 != 234.38333333333335
10004 mean 5198dedb8f03a0003b000001 0.233333333333334 != 0.2333333333333337
10004 score 5198dedb8f03a0003b000001 109.923782370952 != 109.92378237095242
10004 total 5198dedb8f03a0003b000001 2.33333333333334 != 2.333333333333337
10007 avatar 519983d72c4b5c000b000001 /assets/team-avatar.png != /uploads/team/avatar/519983d72c4b5c000b000001/logotipo_bluesoft_alta_resolu%C3%A7%C3%A3o_vertical.png
10007 mean 519983d72c4b5c000b000001 2.96153846153846 != 2.961538461538462
10007 score 519983d72c4b5c000b000001 130.904680792925 != 130.90468079292515
10007 total 519983d72c4b5c000b000001 38.5 != 38.50000000000001
10014 account 519b39331451d7000a000001 The account was not migrated.

<snipped because GitHub can't even handle a 10K line Gist much less a 10K+ line comment>

cc @seuros

@seuros
Copy link
Contributor

seuros commented Oct 15, 2014

  • Fix accounts with invalid stripe keys before migrating (@just3ws)

@seuros
Copy link
Contributor

seuros commented Oct 16, 2014

Can you rebase ?

just3ws and others added 20 commits November 4, 2014 11:44
@just3ws
don't rehash the script/ide startup
0fb8fba
@seuros @just3ws
💀 mongodb
118c24b
@just3ws
WIP: Verifying that the teams are migrated correctly
6c6ea3d
@just3ws
Continuing to verify
fcccc1f
@just3ws
Verify Team#account and Team#account.plans migration
f40ff78
@just3ws
More verfications. Needed to mark the source for the follower class
4e0a2df
@DaneLyons @just3ws
Retina favicons, #270
adad96c
@DaneLyons @just3ws
retina favicons
b3beec7
@just3ws
Bumped gem
a16be40
@just3ws
Script to generate a new box
7a89d9a
@just3ws
Update CONTRIBUTING.md
be6128a
@just3ws
Use Airbrake.io
4b8ca84
@seuros @just3ws
💀 mongodb
6607d51
@just3ws
WIP: Verifying that the teams are migrated correctly
539981a
@just3ws
Added tasks copied from ASM to make downloading the production data e…
a3d5275 
…asier
@just3ws
Updated Vagrant configuration to include cachier and apt dependencies
5fc24d5
@just3ws
Tweaked the precision on the calculated scores for conversion from Mo…
192367d 
…ngoDB => Postgres
@just3ws
Touched Gemfile.lock and Rakefile
d08c40c
@just3ws
Less noise from logs
9dd9121
@just3ws
Updated the Avatar migration to move over the image name and set the …
aa5d18b 
…store dir to fall back to the legacy mongo_id if it's present
@just3ws just3ws force-pushed the replace_mongodb_with_activerecord branch from 31a47df to aa5d18b Compare November 4, 2014 23:37
just3ws added a commit that referenced this pull request Nov 10, 2014
@just3ws
Merge pull request #227 from assemblymade/replace_mongodb_with_active…
c18713c 
…record

MongoDB->Postgres Migration (Part 1 of 2): Migrate team data from Mongoid to ActiveRecord
@just3ws just3ws merged commit c18713c into master Nov 10, 2014
@seuros seuros deleted the replace_mongodb_with_activerecord branch February 16, 2015 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@just3ws @seuros @DaneLyons