Real-Time Data
Warehousing
(presentation)
The following
information protected by USA and international law. Any usage of this
publication or its part can be allowed per request. Consultation available -
leave your email address as a comment.
Foreword
Starting from
five years ago, when I integrated OLTP and Data Warehouse in real time, I more
and more realized advantages of this approach. Discussions of the approach
clarified that success of the real-time Data Warehouse implementation depends
not only on efforts of IT specialists, but primarily on the readiness and
desire of a Business Managers to have a Real-Time Data Warehouse and take
advantage of it in business management.
In order to
move forward in the popularization of the Real-Time Data Warehousing approach,
I prepared a presentation that could be discussed with both IT professionals
and Business Managers.
Description of the Slides
Slide 1
The slide sets the tone of the presentation: a simple and easy explanation of the main functions and advantages of the data warehouse in real time.
The adventure story with the transaction in its journey through the various applications, illustrated with paintings, is an attempt to visualize the presentation and touch the imagination of the audience.
The word Warehousing was used, because the subject of the presentation is not only the Data Warehouse, but also the up- and downstream applications.
To emphasize various aspects of the Real-Time Data Warehousing, the presentation is built in the form a question-answer.
Sites, photos from which are used in the presentation.
Slide 2
The slide is the setting of the presentation purpose: it shows an example of a traditional data processing system using a Data Warehouse.
Simple calculations show that so-called “Big Data Problem” in many cases is the result of accumulation data for further periodic transmission and processing of the data.
The slide shows that the transaction-fish can swim directly from one application to another without delay and lists examples of how the applications can interact in real time.
The One-COMMIT method, when a transaction is loaded both into OLTP and DW under one COMMIT, let both of these applications always be synchronized, because interruption the transaction in the middle, when it is loaded into one application but not yet loaded into another, starts the ROLLBACK mechanism and both application will be synchronized with each other. Using another method of real-time communication requires the creation of a mechanism that implements the rollback.
Slide 4
The slide shows the way of calculation in real time – the usage of Incremental Functions. An Incremental Function is a function that, unlike a regular function in a computer library, that processes all transactions accumulated for a certain time (for example, a month or a year), processes only the data from the current transaction and the result of calculating the Incremental Function from the previous transaction.
Two examples show that an Incremental Function can be calculated based not only on the value of the calculated transaction attribute, but also on other transaction attributes, for example, the transaction number (for the Incremental Function AVERAGE).
For more information on Incremental Functions, see “Doing better real-time calculations in the data warehouse”: https://www.information-management.com/opinion/doing-real-time-calculations-in-the-data-warehouse.
Slide 5
The slide shows the types of data models that can be used for certain types of applications.
Using the Data Vault data model for both OLTP and DW allows you to improve performance of transaction transfer from OLTP to DW without restructuring it, since both applications use the same data model.
The Data Vault model, which has both a normalized structure and specific attributes of the star-schema, also makes it easy to convert transaction data from a normalized structure to a star-schema data model.
All types of data models allow you to process data in real time, and transaction-fish can easily swim from one application to another without any delay.
Slide 6
The slide shows two innovative functionalities that allow Business Management to build a new, more efficient business process using the advanced Real-Time Data Warehousing systems: a real-time data conveyor and current analytical data.
The real-time data conveyor allows you to receive real-time data from any of your applications, and the data will be consistent with each other in all applications.
A valuable achievement of the Real-Time Data Warehousing is that the analytical data, which is traditionally processed periodically on the basis of accumulated transactions, can also be calculated in real time. This allows you to build dynamic and efficient process of Business Management.
Slide 7
The slide shows where and how the Real-Time Data Warehousing can reduce expenses and increase profits from the business.
It is important that the Real-Time Data Warehousing is not only less expensive (there is no cost to create and maintain storage for data accumulation), but also allows you to reduce losses and increase profits from the use of data in real time.
The profitability of the Real-Time Data Warehousing is the strongest argument in discussion of its creation with Business Management.
Slide 8
The slide summarized the advantages of the Real-Time Data Warehousing after explaining the results that began with the goals and explained the ways and benefits of creating it.
Slide 9
The Hamlet’s question addressed both to IT specialists and Business Management.
The role of IT specialists is not only to build the Real-Time Data Warehousing, but first of all convince Business Management that the new approach is more efficient than the existing one.
Creating a library of basic incremental functions is shown as a separate item, because Incremental Functions, as a calculation method, were created specially for the Real-Time Data Warehousing. They do not exist and are not supported by computer libraries used by compilers, and their creation is the start point for building the Real-Time Data Warehousing.
The role of Business Management will increase in comparison with the initial agreement on the new approach, having come to intensive development of new methods of business management operating in real time.
Slide 10
The slide highlights the main purpose of creating the Real-Time Data Warehousing – to help improve the efficiency of Business Management.
Slide 11
The slide is traditional: let listeners ask their questions.
