An Iterative Approach to Data Migration

Data Migration is the process of moving required (and most often very large) volumes of data from existing systems to new systems. I found that such projects can benefit from the use of a tailored development process. This process I came up with can be broken down into three phases.

      Requirements and Design Phase (Mostly requirements gathering activities)

a.      Scope Requirements – A basic understanding of what needs migrated should be obtained. Identify all source and target tables/columns in the migration.  Obtain client signoff on the scope of the migration.

b.     Detailed Requirements – Rules for the data migration should be gathered here. Obtain client signoff on the detailed requirements of the migration.

c.      High Level Design - A light weight design should be created to help divide and conquer

Iterative Development Phase (Mostly development and migration activities). These steps should be repeated as long as there are defects or requirement changes.

a.      Refine Requirements - Identify requirement gaps and modify

b.     Develop/Modify – Modify scripts to reflect requirement changes and defects

c.      Test Migration - Extract Data to the test database

d.     Test/Verify – Identify defects and requirement change requests  

       Production Phase - This phase includes the final migration and post migration activities.

 a.   Production Migration - Extract data to the production database

b.      Test – Identify errors, and the need for data cleanup

c.       Perform Data Cleanup – Perform manual steps for data cleanup