Since 2003, the consultancy and analyst company Passionned has been closely monitoring the market for both ETL and Data Integration tools. In the past, the focus was on the well-known market leaders who were often regarded as visionaries and leaders. Many organisations used to assume that they had automatically made the right choice if they purchased a tool from one of these companies.
Since the late nineties, however, the market has changed substantially. Practically all the major BI vendors have purchased or developed their own ETL tools. Given that a central data warehouse is one of the major cornerstones of a successful BI solution, their choice to move into the ETL marketplace is understandable.. After all, 70-80% of the costs involved in a successful BI system are infrastructure, creating reliable ETL processes and a well designed method of keeping the data warehouse current is essential.
Survey motivation and set-up
The first reason is that the focus of many organisations was on BI tools for end users, such as OLAP reporting and dashboards, while ETL, Integration and Data Warehouses form the cornerstones of a successful BI solution. The time and energy spent on designing ETL processes correctly and selecting a matching ETL and data integration tool will pay dividends in the future. Thomas Siebel, the founder of Siebel, (now Oracle) was quoted in an interview: “Organizations may save a lot of money when valuable data stored throughout the organization in seperated silo’s can be integrated and combined”.
The second reason to evaluate and promote ETL tools, is that many organizations still build their data warehouse by hand, writing complex SQL and stored procedures. Many organisations still find themselves in the ‘stone age’. Data warehouses are still being developed manually using either SQL or PL/SQL, while developer productivity would be increased by a factor of three to five if a proper ETL tool was used. Additionally, the reliability and stability of data warehouses built using an ETL tool is much higher, since more aspects can be checked and monitored in relation to each other, metadata being a case in point. It should, however, be noted that merely using an ETL tool does not automatically guarantee success.
Thirdly and finally: the market for Business Intelligence tools is growing again with expectations of large revenue increases this year for some of the suppliers, but there appears to be no increase in the success rate of the warehouses and BI systems that we are building. Despite the availability of better and better business intelligence software at lower and lower prices we are not producing better systems for our users. Our mission is to find the factors that make an ETL project a success.
How the ETL Tool Survey 2012 is carried out
- A list of 90 criteria was developed as a basis for the comparison, using input from the developer community and focussing on characteristics that play a major role in the quick development of a successful BI infrastructure.
- Passionned conducted in-depth interviews with all the vendors listed on the ‘List of ETL tools’ page regarding their ETL solutions.
- In addition, vendors were given the opportunity to provide a live, two-hour demonstration in order to present the functionality of their ETL tool. PowerPoint presentations were avoided as much as possible. During these demos, Passionned evaluated whether the tool was able to handle particular situations by means of Q&A sessions. Where there was still doubt, Passionned gave the tool either zero or half a point.
- Vendors were asked to provide list prices for their tools by an official Request for Proposal (RFP), with two configurations (see also the criteria). This only gave a price indication since all the products are priced in different ways.
- In many cases an intensive dialog took place between Passionned and the vendors to obtain answers to outstanding questions. Ambiguous or misleading answers were not accepted. All answers were confirmed in writing by the suppliers to avoid misunderstandings.
- Finally, a number of developers from end-user organisations as well as BI consultants provided feedback with regard to the results of the ETL study. Whenever the necessary, further investigation took place and the evaluation was altered accordingly.
