Four years ago, when we published the second edition of this survey, we saw that not many ETL tools had good, reliable functionality for real-time application integration (EAI) projects. Since then, many ETL tools have added tools for real-time extraction, transformation and integration, and there has been an almost complete convergence between ETL and EAI tools into a new market which is being called Data Integration.
Best of both worlds: a marriage
A marriage? It is clear that the vendors think that this is where the future lies, but when asked most will admit that they have very few customers yet who are using what amounts to real-time ETL. In the table below it is clear that real-time systems and traditional ETL come from very different backgrounds, but there are examples where combining the two produces the best of both worlds.
Likely to merge completely
It is clear that many of the suppliers have now integrated these tools, but ETL and EAI tools still satisfy different customer needs, each market will probably continue to exist on its own, and have its own user population mainly because that is what the majority of the customers still want. What we have seen is that these tools increasingly make use of a common metadata layer, a repository of transformations and business logic and standard adaptors to connect to a wide variety of systems.
Data is entered only once in the enterprise
Finally, the nature of ETL for Business Intelligence purposes, differs in many aspects from the nature of EAI. ETL is about moving, transforming and cleaning large amounts of data not more than a few times a day from many sources into one place – convergence. Whilst with EAI we often move realtively small amounts of data, transactions, spreading them across various systems – divergence. We use ETL for data migration, for one view of the truth or data integration to allow better decision making. In general we use EAI for process optimization and workflows and to make sure that data is entered only once in the enterprise, although some companies are moving towards real-time Business Intelligence and using EAI tools to keep the data warehouse up to date.
ETL and EAI have a strong relationship
Marriage? Yes. From the vendors point of view, IBM, Microsoft, Oracle and other vendors in the EAI and ETL market, are producing a unified platform, which makes it look as if the products will at least be living together. Recently we have seen acquisitions where integration and business intelligence tools from companies like BusinessObjects, Cognos and Datamirror have been bought by platform vendors, with the consolidation in this marketplace many of the suppliers are no longer dependent on one product for their revenue, so as long as the customer keeps using their revenue generating product, which may be a database or a particular brand of front end software (like MS Office) then the costs of the ETL and the broker software will continue to be reduced.
In our opinion ETL and EAI have a strong relationship, should use the same business rules and definitions, but have a different architecture – perhaps it looks like a marriage after all!
Strengths of each: ETL vs EAI:
- Excel at bulk data movement & batch data integration
- Provide for complex transformations, aggregation from multiple sources and sophisticated business rules.
- Assume considerable data delays.
- Are batch-oriented, making them fast and simple for one-time projects and testing
- Offer little in the way of workflow
- Work primarily at the session layer
EAI: Enterprise application integration systems
- Are limited in data movement capabilities
- Offer less sophisticated transformation and extraction functions
- Operate in real time
- Work better with continuously interacting systems
- Are workflow-oriented at their core
- Work primarily at the transport layer