We are closely monitoring the ETL market
Since 2003, the consultancy and analyst company Passionned Group has been closely monitoring the market for both ETL and Data Integration tools. In the past, the focus was on the well-known market leaders who were often regarded as visionaries and leaders. Many organisations used to assume that they had automatically made the right choice if they purchased a tool from one of these companies.
The market changed substantially
Since the late nineties, however, the market has changed substantially. Practically all the major Business Intelligence vendors have purchased or developed their own ETL tools. Given that a central data warehouse is one of the major cornerstones of a successful Business Intelligence solution, their choice to move into the ETL marketplace is understandable. After all, 70-80% of the costs involved in a successful Business Intelligence system are infrastructure, creating reliable ETL processes and a well designed method of keeping the data warehouse current is essential.
Survey motivation and set-up
- The first reason is that the focus of many organisations was on Business Intelligence tools for end users, such as OLAP, reporting and dashboards, while ETL, Integration and Data Warehouses form the cornerstones of a successful Business Intelligence solution. The time and energy spent on designing ETL processes correctly and selecting a matching ETL and data integration tool will pay dividends in the future. Thomas Siebel, the founder of Siebel, (now Oracle) was quoted in an interview: “Organizations may save a lot of money when valuable data stored throughout the organization in separate silos can be integrated and combined”.
- The second reason to evaluate and promote ETL tools, is that many organizations still build their data warehouse by hand, writing complex SQL and stored procedures. Many organisations still find themselves in the ‘stone age’. Data warehouses are still being developed manually using either SQL or PL/SQL, while developer productivity would be increased by a factor of three to five if a proper ETL tool was used. Additionally, the reliability and stability of data warehouses built using an ETL tool is much higher, since more aspects can be checked and monitored in relation to each other, metadata being a case in point. It should, however, be noted that merely using an ETL tool does not automatically guarantee success.
- Thirdly and finally: the market for Business Intelligence tools is growing again with expectations of large revenue increases this year for some of the suppliers, but there appears to be no increase in the success rate of the warehouses and BI systems that we are building. Despite the availability of better and better Business Intelligence software at lower and lower prices we are not producing better systems for our users. Our mission is to find the factors that make an ETL project a success.
How our survey is carried out
- A list of 90 criteria was developed as a basis for the comparison, using input from the developer community and focussing on characteristics that play a major role in the quick development of a successful Business Intelligence infrastructure.
- Passionned Group conducted in-depth interviews with all the vendors listed on the ‘List of ETL tools’ page regarding their ETL solutions.
- In addition, vendors were given the opportunity to provide a live, two-hour demonstration in order to present the functionality of their ETL tool. PowerPoint presentations were avoided as much as possible. During these demos, Passionned Group evaluated whether the tool was able to handle particular situations by means of Q&A sessions. Where there was still doubt, Passionned Group gave the tool either zero or half a point.
- Vendors were asked to provide list prices for their tools by an official Request for Proposal (RFP), with two configurations (see also the criteria). This only gave a price indication since all the products are priced in different ways.
- In many cases an intensive dialog took place between Passionned Group and the vendors to obtain answers to outstanding questions. Ambiguous or misleading answers were not accepted. All answers were confirmed in writing by the suppliers to avoid misunderstandings.
- Finally, a number of developers from end-user organisations as well as Business Intelligence consultants provided feedback with regard to the results of the ETL study. Whenever the necessary, further investigation took place and the evaluation was altered accordingly.