Skip to main content

Migrating from MS SQL Server to MongoDB

The following contains notes from various attempts at migrating 2.5GB of MS SQL Server data to MongoDB, on an 8 GB, quad-core, 64-bit Windows 7 Enterprise machine.

  1. [TERMINATED] http://mssql2mongo.codeplex.com/ : Simple to use. Exponential slowdown observed in migration. For a total of the 50 million records spanning two tables, the following migration times were observed: 1 second for the first 100,000 records,  30 minutes for 1,000,000, 20 hours for 16 million (after which I terminated the process).
  2. [DID NOT WORK] http://rubydoc.info/gems/mongify/ : A ruby-based approach. Use Ruby 1.9.3 (tiny_tds dependency causes problems with Ruby 2.0). Install DevKit before installing the mongify gem. Also, use 'sqlserver' as the adapter in the .config file. Then, before running 'mongify check <config-file>', install the gems activerecord-sqlserver-adapter and tiny_tds. At this stage, it fails. 'mongify translation <config-file>' fails as well.
  3. [MODIFIED/WORKED!!] http://mssql2mongo.codeplex.com/ :
    1. introduced concurrency (since both SQL SERVER read and MongoDB write support it!)
    2. tweaked and customized page size for each table (trial-and-error)
    3. This version works great -- about 45 minutes to migrate the whole data.
    4. Note: Ensure that indices are created on the primary keys of all the tables.
    5. Only one source file is changed (see patch)


Comments

  1. UnknownJanuary 27, 2022 at 5:16 AM

    Hey team I want to know some details about mongify the following are my question :
    1 . How to ensure the data is not corrupted while migrating ?
    2. data types conversion from sql to mongo
    3 . Check what permissions are required for Mongify to work ?For example, the environment in the Bank is restricted
    4 . What is “check” doing ? (Mean it only checks the connection or checks the schema also)
    5 . How is Mongify working ? What is the mechanism ?
    6 . Constraints – if any for mongify – like linux / ubuntu / versions.

    ReplyDelete

Post a Comment

Popular posts from this blog

the brave new economy

A free(er) market is emerging and has been emerging ever since the advent of the Internet and the Web. Newer technologies have accelerated this. Old inefficiencies in knowledge and access to the market are quickly disappearing. Participation is increasingly peer-to-peer (P2P). Participants can produce, create, curate, give, lend, sell, share, auction, consume, take, borrow, buy, reuse, rent and/or barter goods, services, cash, credit, currency, equity, debt and/or knowledge. The resultant economy is highly collaborative and is sometimes referred to as the ‘ sharing economy ’. Efficiencies are being introduced and value is being created in the smallest of markets. Non-profit activities are thriving as well, in addition to for-profit ones. The marketplace is becoming, at once, global and local. An important subset of the new economy is collaborative and crowdsourced , and may be described as commons-based peer production . Asymmetries in information are diminished by algorithms (Google
Post a Comment
Read more

the World Cup draw -- an alternative proposal

The FIFA World Cup seeding and draw system has consistently produced groups of death and groups comprised mainly of teams that fail to invoke passion or imagination from fans around the world, except perhaps for such teams' own fans. The latest edition is no exception -- and may even be the worst ever on this measure. An outline of the draw system can be seen here . We have one group which contains Switzerland, Ecuador and Honduras and another consisting of Russia, Algeria and South Korea!! And, on top of this, we have at least three, if not four or more, groups of death. Depending on one's perspective, esp. given the recent rise of African powerhouses, all of the rest of the groups may even be considered Groups of Death!! This is a disservice to football fans around the world, not to mention the skilled and enterprising teams that worked hard to get here. To address this, I propose a simplified draw system : Randomize Pot 1 and assign to Groups Assign remaining Pots t
4 comments
Read more