The numerous challenges faced during one of my project's data migration phases was duplicate records. Many hours were spent finding and subsequently removing the duplicate records. No proven methodology was ever employed to cleanse the data.
For future migrations, I suggest using Data Cleansing approach to clean data. Data Cleansing is the process of (semi-)automatically detecting and correcting errors in data collections. It differs from Data Validation in that validation means data is rejected by system at entry and is performed at entry time, rather than in batches of data as with migration. Data Cleansing is also synonymous with the term data scrubbing.