Scalability of data

The financial services industry is known for its large data volumes. The need for detailed customer information and extensive financial analysis, the consideration of risk management, regulatory changes, and fragmented liquidity has led to an explosion of (market) data volume.

There is also the fact that all institutions in the financial services have to deal with very sensitive data. Detailed information on credit card holders and their transactions, the medical history of health insurance policy holders, the financial situation of clients (e.g. in wealth management) are just a few examples of the confidentiality of the data these organizations are dealing with on a daily basis. That is why the demand for highly sophisticated security mechanisms is a common theme in this industry.

A third determining factor for financial services is performance. Trading departments need real-time information on the stock market; a hedge fund manager wants to follow up on the performance of his fund with the ability to analyze deeper to understand the cause of changes; the CRO requires quick information to manage the risk and minimize the expected loss; a financial analyst in an insurance company is interested to find hidden patterns in the customer data helping him with new business opportunities.

Thus, financial institutions require a scalable solution that can cope with large data volumes, ensure the right level of security and can handle the magnitude of requests from stakeholders and a growing user community

These needs resulted in huge investments in IT architecture to manage the amount of data and provide the right framework. One of my customers, one of the biggest banks worldwide, had ordered hardware with multiple Petabyte of disk space. Their aim was to report the consolidated bank at the detailed financial instrument level on a daily basis. The data load for this kind of endeavor into a data warehouse (DW) had to be optimized with sophisticated sort algorithms in order to provide the data on time. Once the data was in the operational data store of the DW the data had to be cleansed, enriched and then loaded into a data mart that is optimal for reporting.

While the development of a well defined data warehouse is a good concept and worth doing, in this particular instance the time between the transactions took place and their reflection in a report was just too long. For the daily group consolidation report it was sufficient, but only if the intercompany transactions were matching. If not, an exception report was produced and someone had to check all payables and receivables where the relations were not matching. Once this manual review was finished, the corrected results were entered into the enterprise resource planning system (ERP), which then triggered an update of the DW and the following processes.

That is not optimal for ad hoc reporting and definitely not a solution for some of the demanding business requests mentioned above (time is money). As such a reporting tool that can access the transactional data directly, combining it with the information from the other sources and presenting the information with the help of an integrated metadata layer is preferable.

Depending on the role of the business user it is not necessary to have always the full low level detail visible in a report. The CFO for example wants to get a quick overview of the business. An aggregated dashboard with the key performance indicators that are relevant for the CFO will work. Wherever a more thorough analysis is required, he can look at the KPI from different angles (e.g. segments, products, channels), just by clicking on a different tab of the dashboard. If that is still not detailed enough a deep-dive into the transaction report directly from the dashboard is possible. We have implemented that multiple times and it is always remarkable how well received the variety of visualizations, the flexibility of report development, the ease-of-use, and the scalability of the reporting tool is. The ability to drill anywhere with great performance, even when handling enormous data volumes is essential for the business to become as efficient as possible and is a great competitive advantage!