Skip to main content

Migrating from MS SQL Server to MongoDB

The following contains notes from various attempts at migrating 2.5GB of MS SQL Server data to MongoDB, on an 8 GB, quad-core, 64-bit Windows 7 Enterprise machine.

  1. [TERMINATED] http://mssql2mongo.codeplex.com/ : Simple to use. Exponential slowdown observed in migration. For a total of the 50 million records spanning two tables, the following migration times were observed: 1 second for the first 100,000 records,  30 minutes for 1,000,000, 20 hours for 16 million (after which I terminated the process).
  2. [DID NOT WORK] http://rubydoc.info/gems/mongify/ : A ruby-based approach. Use Ruby 1.9.3 (tiny_tds dependency causes problems with Ruby 2.0). Install DevKit before installing the mongify gem. Also, use 'sqlserver' as the adapter in the .config file. Then, before running 'mongify check <config-file>', install the gems activerecord-sqlserver-adapter and tiny_tds. At this stage, it fails. 'mongify translation <config-file>' fails as well.
  3. [MODIFIED/WORKED!!] http://mssql2mongo.codeplex.com/ :
    1. introduced concurrency (since both SQL SERVER read and MongoDB write support it!)
    2. tweaked and customized page size for each table (trial-and-error)
    3. This version works great -- about 45 minutes to migrate the whole data.
    4. Note: Ensure that indices are created on the primary keys of all the tables.
    5. Only one source file is changed (see patch)


Popular posts from this blog

Diversity in politics

There are so many strains of thought and ideology in secular democracies around the world, exemplified, perhaps best, in contemporary American political life. Some of these strains of thought are compatible with others and may even be subsets of others. Many of them are violently opposed to others. In the American context, we see, at least, the following :  environmentalismcultural progressivismeconomic socialismprogressive-lite Keynesian welfare + neocon warfare hawks = the establishment / moderates / centristscivil libertarianismnon-interventionismsocial conservatismConstitutional conservatismfree-market libertarianismpopulism, protectionism & nativism, anti-progressivism, anti-establishment
A neat left-right divide no longer makes sense. Most people find themselves on a spectrum between very different political philosophies and emphases.
The two major political parties represent uneasy marriages that exist because for historical reasons or current tactical alliances. The fragil…

The moral filth of neo-liberal and neo-conservative statism

or, progressive socialism is not the greater evil.
Introduction
After years of trying to make sense of how the world works, economically and politically, and how, according to some, it ought to work, I have increasingly come to the realization that the libertarian political-philosophical persuasion is the only one consistent with justice. A just society must necessarily be a free society. Specifically, one that is founded on libertarian principles -- private property, NAP, non-initiation of violence, easement & homesteading rights, etc. In fact, the “justness” (righteousness) of a society will increase in concert with and to the degree to which a society moves from totalitarian statism to democratic statism (of the neo-conservative or neo-liberal varieties) to minarchism to anarchism.
After spending a few zealous but misguided and ignorantly happy years occupied with mainstream GOP politics, including loyally and foolishly supporting the Bush-Cheney-Rove regime, despite serious mo…