Unlike SOA, which can support real-time data movement, data integration (typically) provides adequate business information (data replication) without up-to-the-minute access of information. In many cases, the data is weeks, even months, old, and the data mart or data warehouse is updated through antiquated batch, extract-aggregate-and-load, processes. Indeed this is the way integration is done today, using a data integration product or in most cases no product at all. When I was doing research for my last book, for instance, I found that FTP was still the primary form of data integration.

Things are changing, fortunately. SOA, and the technology that comes with it, allows data warehouse architects and developers to move information – no matter where it comes from or where it is going – as quickly as they want to move it. As a result, it is not unheard of to have all participating databases and services in a SOA solution receiving new data constantly, thus providing more value to those using the source and target systems existing within the SOA – including those who use them as a data warehouse or data mart.

Therefore, the rise of SOA will also lead to the rise of real-time data warehouse solutions, or could replace the notion of data warehousing all together. For instance, we could place the data behind services (data services or abstract data services), with users able to leverage up-to-the-minute information to make better business decisions using BI tools or BAM, or just snap them into composite applications.

As fore mentioned, as time goes on the data integration products may not be needed as SOA architects craft services that abstract both operational and aggregated data, in some cases leveraging aggregated data without having to replicate and change the data, but instead do it through abstraction layers. This approach, if possible for your domain, is much less costly and less complex. In essence, the SOA becomes the place to leverage services that can deal with the data layer(s) through many types of abstraction services, services that can be mixed and match in composite to create solution instances for the SOA. Thus, the value of SOA, bringing all of these things together as a platform for business solutions.

So, how will we get there, and what can the data integration vendors do? I'll hit that later in the week.

Posted by Dave Linthicum on July 11, 2006 04:28 AM