Development prospects for Ranet OLAP Technology using Artificial Intelligence

One of the main areas of development for business intelligence systems today is the integration of artificial intelligence functions. Industry experts see great potential in using machine learning and AI algorithms in business intelligence systems, especially in such areas as retail, finance and banking, telecommunications, insurance and healthcare, when it becomes necessary to analyze historical data for several years or decades.

In this article, we would like to share our vision of further development of our Ranet OLAP solution using artificial intelligence.

Currently, Ranet OLAP is an innovative technology and rich data processing and business analysis tool that is essential for making fast and informed management decisions. At the core of this technology are:

• storage and data marts,
• OLAP cubes developed for typical tasks of analyzing financial and economic activities, typical of each organization,
• data of OLTP-systems (ERP, etc.), in which business transactions and related primary documents (payment and cash, invoices and etc.), financial and business transactions are processed.

Business configurations can be supplemented with data from CRM, CPM(BPM) and other systems that are involved in automating end-to-end business processes of an organization and provide critical information for analysis and management decision-making.

The product has been successfully commercialized since 2014.

It should be noted that the OLAP technologies we use, implemented in an open-source columnar analytical DBMS (allowing you to perform real-time analytical queries on structured big data) and Mondrian, use separate elements of artificial intelligence for analysis tasks at the level of software platforms. However, we plan to develop our solution into a full-fledged platform for serving and managing data using AI.

Here are the main features we are going to focus on:

• Integration with high-performance OLAP engines (Pentaho Mondrian, ClickHouse and others), which are perfect for machine learning;
• Using very fast data sources:
  • • ClickHouse – columnar database, operates with standard SQL queries;
    • Pentaho Mondrian - ROLAP, MDX queries to multidimensional cubes and others;
• Working with huge arrays of historical retrospective data:
  • • support for machine learning algorithms;
    • The MDX query language, focused on accessing multidimensional data structures, has many statistical functions, functions for working with time measurement, rank calculations, data refinement, etc. to build forecasts using artificial intelligence;
    • professional intelligent report generators based on OLAP-cube technology;
    • search for patterns in data changes and forecasting (multiple calendars, parallel periods, divergent horizons, cumulative totals, scenarios), the ability to use Data Mining technologies.
• Domain-oriented storage and Data Marts:
  • • Hierarchically organized data (Drill Down/Up operations, etc.) in business terms;
    • Ready-made data marts (sales, purchases, stocks, finance, debt, personnel, etc.) for data integration of OLTP systems (ERP, HCM, EAM, CRM, etc.);
    • Support for performance indicators (KPI).
• Built-in visual designers of interactive analytical reports, with visualization of results in pivot tables and charts:
  • • tools focused on business users;
    • dynamic models for building interactive analytical reports;
    • built-in algorithms for parsing and dynamic generation of MDX queries;
    • report templates that implement built-in methods of economic analysis: ABC analysis (Pareto principle), XYZ analysis (coefficient of variation), integrated ABC / XYZ analysis, comparison and analysis of deviations, objective rating analysis, cross-sectional analysis, various samples;
    • report templates that implement built-in methods of analysis such as “What-if” and predictive analysis.
• Operations on the time axis:
  • work with time measurement;
  • multiple calendars;
  • diverging horizons (time axis beyond the horizon for distant future periods);
  • calculated temporary members and aggregates;
  • periods: parallel, open, close;
  • comparison by periods, calculation of trends.
• Multidimensional calculations:
  • calculated dynamic expressions and members;
  • aggregate and statistical functions, logical functions and complex scenarios, IIF function;
  • formatting the result (heat maps).
• Pre-calculation and real-time calculations:
  • preliminary calculation of aggregates for cube data slices, this ensures the speed of processing typical requests of thousands of users (calculations are available for everyone, and are not made for each user when processing a request);
  • monitoring of long-term user requests allows you to identify data slices for which it is necessary to organize preliminary calculations.
• The speed of analysis with the growth of data in a geometric progression:
  • Methods of preliminary processing (calculation) of the cube, which stabilize the response time to a user's request to multidimensional data in the interactive analysis mode at the level of seconds.
• Reducing the cost of data storage and automation of operations:
  • • industry standards and the ability to create composable data and analytics architectures with a view to decades;
    • Opening a new line of business will no longer require revising the architecture of business intelligence solutions, expanding the team of IT specialists (data engineers);
    • planning of cloud ecosystems of data and analytics.

 

In addition, the project plans to implement an intelligent mechanism for processing data queries in the MDX language:

1. Any MDX query, including those manually developed by the user, will support data drill down commands (an Excel pivot table can only contain an automatically generated query);
2. Automatic detection and identification of slices of detailed data separately for each of the elements selected by the user (Excel will open all nodes) and generation of corresponding MDX queries for them;
3. Ability to emulate user roles to access cube data without the need to administer them on the OLAP server. Even the most advanced user will not get access to “foreign” data. This is extremely important for business intelligence cloud services serving thousands of users (Multi-Tenancy);
4. Advanced object models for describing economic analysis methods without restrictions on the number of indicators (indicators), with dynamic formatting of the result (including heat maps), interactive filters and automatic generation of highly efficient MDX queries based on them.

 

Analytic report templates provide advanced mining capabilities directly in the OLAP Browser dynamic pivot table and implement:

1. Multi-factor models with dynamic classification of customers, product range, etc., and taking into account changes over time; Suitable for any fields of activity;
2. Normative models for scenario (forecast-plan-actual, etc.) analysis; The ability to operate with standard values (indices) that change over time, in various sections (for example, to optimize costs and improve the efficiency of the organization); Expert analysis (the ability to receive and integrate expert assessments);
3. Cluster analysis (segmentation by customer groups, product range, etc.);
4. Analysis of the influence of the relationship of factors (for example, analysis of the dispersion of price and volume (Price Volume Mix);

5. Forecasting based on facts: "Naive" forecasting model ("tomorrow will be like today", but taking into account some indices; Exponential (Exponential Moving Average, EMA) and Simple (Simple Moving Average, SMA) moving averages in the time window;

AI-based regulation monitoring service

The plans for the development of our Ranet OLAP technology involve the construction of a service for monitoring the regulations of document processing workflows based on Artificial Intelligence.

Idea

Creation of a service that allows monitoring the regulations of work processes for processing primary business and financial documents and, based on analysis using artificial intelligence, generate "smart recommendations" for the user of the System.

Application area

The service must be able to:

• analyze the fulfillment of the regulatory deadlines for processing documents (behavioral analysis) and, on the basis of AI, form the image of the user (client, counterparty) of the System;
• issue reminders and hints to the user of the System (including the timing of work processes);
• analyze the workload by categories of users of the System and the level of executive discipline.

Machine learning methods are applied to Business Configurations.

Novelty of the idea

Diverse and in-depth analysis of the regulations for the fulfillment of obligations, streamlining work processes.

Perspective

The introduction of this service into the operational document flow of the financial and economic activities of a commercial organization will have a significant impact on the formation of a policy for working with clients, product groups, etc.

When implementing the service, typical machine learning tasks related to prediction, classification, and anomaly detection will be affected.

In addition, based on the implementation of the idea described above, it will be possible to develop and implement other related services in the business processes of the organization.

Competitivity

Working with regulations and according to regulations is one of the key services in the business processes of any organization. The Service will give “smart recommendations” to System users when making decisions or, conversely, determine patterns of behavior.

In addition, the service can be adapted to any areas of activity where work according to regulations is important. For example, the Asset Management System (EAM), which tracks routine maintenance processes for equipment.

A small summary

According to a survey conducted by the media resource MIT Technology Review, in three years artificial intelligence will cover at least a third of the main business processes, including the field of business intelligence. This is how 30% of surveyed companies and 41% of financial institutions assess its prospects.

Based on our own many years of experience and deep expertise in the field of data processing and analysis, we believe that in the era of Big Data, the use of artificial intelligence will be the next logical step in the development of business analysis systems.

Augmented analytics is, in fact, one of the tools to increase the level of literacy and culture of working with data. With the help of built-in machine learning tools, query processing in a user-friendly language in business terms, more and more people begin to expect from BI systems not ready-made canvas reports and difficulties in their development, but interactivity and democracy.

Some experts are already of the opinion that classical BI is either “post-mortem analysis” or an attempt to discern patterns in a huge data set, which is incredibly difficult without the use of artificial intelligence and machine learning technologies. The main goal is to look ahead, and here one cannot do without the use of modeling or forecasting methods.