{"id":1649,"date":"2012-07-26T18:08:08","date_gmt":"2012-07-26T22:08:08","guid":{"rendered":"http:\/\/www.iri.com\/blog\/?p=1649"},"modified":"2026-01-08T15:28:27","modified_gmt":"2026-01-08T20:28:27","slug":"the-case-for-test-data","status":"publish","type":"post","link":"https:\/\/www.iri.com\/blog\/test-data\/the-case-for-test-data\/","title":{"rendered":"The Case for Safe, Intelligent Database Test Data"},"content":{"rendered":"

Database Test Data Usage<\/strong> – This blog caught my eye\u00a0because of its title, Do the right thing when testing with production data<\/a>. \u00a0<\/strong>It struck me as oxymoronic, since we know production data should not be used for testing at all …<\/p>\n

Of course we know how tempting it is to use production data for testing\u00a0applications, simulating databases, prototyping ETL operations, and just about anything else that needs to work with the real thing. However, it is not worth the legal risk of exposing that data to breaches. And, even if you were certain that you could lock that data down somehow, have you considered its potential inadequacy for testing purposes? If you only develop and test with data that’s real today, what happens when your platform has to work with different data tomorrow? Ideally, you need to be able to stress-test new platforms, and that means taking into account future data changes in values, formats, ranges, and\/or volumes.<\/p>\n

Security Active’s article did, however, acknowledge the data breach risk, and suggested that the right thing to do was to obfuscate the production data with appropriate protection tools and scripts — like IRI FieldShield <\/a>—\u00a0which would encrypt, mask, pseudonymize or otherwise de-identify the production data to make it safe for testing.<\/p>\n

And that is certainly a popular, though I submit not necessarily the best, approach. What happens if a security method is compromised (e.g. the decryption key is learned or the permutation algorithm reversed)? Or what if the\u00a0test data is safe, but ends up looking unrealistic or being referentially incorrect? And how would morphing 100,000 social security numbers in one fact table, for example, help you test an enterprise data warehouse that will need to handle 50,000,000 numbers distributed\u00a0and linked across multiple tables?<\/p>\n

Better if you could generate file, report, or database\u00a0test data<\/a> without using production data at all, but where that test data has:<\/p>\n