I have worked on many digital transformation projects and all of them take one of several paths for maintaining the old data from the legacy system(s). These are:
- Throw out all the old data
- Migrate all the data to the new system(s)
- Migrate summary and or current data
In some projects you move all the legacy application data to the new application. But this is not normally the case. So, what do we do with the data that we don’t move to the new application?
So, what do we do with the old legacy data?
We create a DATA MORGUE, a place where data goes to die
Many times, we have critical data in a system that we know we will need for a short while. As time progresses the data becomes less valuable until eventually there is no value left. So, unlike a data warehouse where we put data with high value, we put data in a Data Morgue that has some value but not a lot of current value.
Equally important is that we put the data into the Data Morgue while someone remembers what that data means.
Over the next few posts I will be explaining the purpose of a Data Morgue, what goes into it, when to use it and how to create one. It is different from a normal data warehouse or data lake. It serves a very specific purpose because when it is created because the data has much more value than what it will have in a few weeks, months or years. And eventually the Data Morgue just dies itself.
What is a DATA MORGUE?
It is the tables and data that you have in the legacy application that won’t be moved to the new application along with the data that will be moved. In most digital transformations the new application has just summary information, starting balance information, recent historical information.
We are not continuing to maintain all the history from the old systems. Many old systems have months and even years of history that we will not be moving to the new system. And in many complete transformations no legacy data is moved to the new way of doing business. I.E. there is a clean break from the old way to the new way.
So why don’t we move all that data? This is for reasons that we all know:
- Data Quality
- Data Completeness
- Data Consistency
- Data is no longer relevant to the way we will be working
- Data is not detailed enough
- Cost of transforming old data
These make it way to expensive, too complex, and in some cases completely unnecessary to migrate all the old data. So, we don’t move all the data. That is why we create a Data Morgue as a better alternative that is much more cost effective. We still may need the data, but we don’t want it in the new system.
Now we have the idea of why a data morgue is an essential tool. But what makes it different?
There are many details, but the concept is that we make the data independent of the application. This does not mean we de-normalize it like a true data warehouse, but we may partially de-normalize it. We don’t leave it in the same format as the full transaction processing system because we want people to be able to find data after the application and the experts for that legacy process are no longer available.
So there are the rules for creating a data morgue and what data do we put in the data morgue but the key to the usefulness of a Data Morgue is that we make the data independent of its originating application with as little work as possible and only preserving the data that will have some value.