Skip to main content

Migrating from MS SQL Server to MongoDB

The following contains notes from various attempts at migrating 2.5GB of MS SQL Server data to MongoDB, on an 8 GB, quad-core, 64-bit Windows 7 Enterprise machine.

  1. [TERMINATED] http://mssql2mongo.codeplex.com/ : Simple to use. Exponential slowdown observed in migration. For a total of the 50 million records spanning two tables, the following migration times were observed: 1 second for the first 100,000 records,  30 minutes for 1,000,000, 20 hours for 16 million (after which I terminated the process).
  2. [DID NOT WORK] http://rubydoc.info/gems/mongify/ : A ruby-based approach. Use Ruby 1.9.3 (tiny_tds dependency causes problems with Ruby 2.0). Install DevKit before installing the mongify gem. Also, use 'sqlserver' as the adapter in the .config file. Then, before running 'mongify check <config-file>', install the gems activerecord-sqlserver-adapter and tiny_tds. At this stage, it fails. 'mongify translation <config-file>' fails as well.
  3. [MODIFIED/WORKED!!] http://mssql2mongo.codeplex.com/ :
    1. introduced concurrency (since both SQL SERVER read and MongoDB write support it!)
    2. tweaked and customized page size for each table (trial-and-error)
    3. This version works great -- about 45 minutes to migrate the whole data.
    4. Note: Ensure that indices are created on the primary keys of all the tables.
    5. Only one source file is changed (see patch)


Comments

  1. UnknownJanuary 27, 2022 at 5:16 AM

    Hey team I want to know some details about mongify the following are my question :
    1 . How to ensure the data is not corrupted while migrating ?
    2. data types conversion from sql to mongo
    3 . Check what permissions are required for Mongify to work ?For example, the environment in the Bank is restricted
    4 . What is “check” doing ? (Mean it only checks the connection or checks the schema also)
    5 . How is Mongify working ? What is the mechanism ?
    6 . Constraints – if any for mongify – like linux / ubuntu / versions.

    ReplyDelete

Post a Comment

Popular posts from this blog

the World Cup draw -- an alternative proposal

The FIFA World Cup seeding and draw system has consistently produced groups of death and groups comprised mainly of teams that fail to invoke passion or imagination from fans around the world, except perhaps for such teams' own fans. The latest edition is no exception -- and may even be the worst ever on this measure. An outline of the draw system can be seen here . We have one group which contains Switzerland, Ecuador and Honduras and another consisting of Russia, Algeria and South Korea!! And, on top of this, we have at least three, if not four or more, groups of death. Depending on one's perspective, esp. given the recent rise of African powerhouses, all of the rest of the groups may even be considered Groups of Death!! This is a disservice to football fans around the world, not to mention the skilled and enterprising teams that worked hard to get here. To address this, I propose a simplified draw system : Randomize Pot 1 and assign to Groups Assign remaining Pots t...
4 comments
Read more

the resurgence of liberty

Political awareness and activism at the grassroots level is rising in the United States. The political spectrum in the United States is changing. Long seen as one-dimensional (left vs. right), there has been a strengthening and orthogonalization of at least one more axis -- that of liberty, and the lack thereof. Libertarianism is most certainly on the rise -- with a rising concern, across all demographics, for broader economic ( laissez-faire capitalism  and Austrian economics ) and personal freedoms ( civil libertarianism ). While a majority of the new libertarians come from the right , there is also growing number of left-libertarians as well as a significant exodus of liberals and progressives toward classical libertarianism. Fiscal conservatism is a defining characteristic of the traditional right-wing. This ideology is generally hawkish on spending, budget deficits and taxes and moderate on business regulation and social safety-net programs, advocating for fiscal and mo...
1 comment
Read more