Keep analytics data updated
Beginner | 15 Minutes
Overview
Analytics dashboards and reports are only useful when the data in them is updated and current.
This tutorial walks you through the process of automating data import and processing in Birst.
📋 Requirements
|
Tutorial
Keeping data fresh can be achieved by scheduling periodic updates. Let's assume that we are extracting data tables from the Infor data lake to be used in Birst reports and dashboards. We have completed creating the dashboards and we want to update the data in Birst, automatically on a scheduled basis.
For this purpose, we make use of the 'Orchestration Workflow', a simple process with a few clicks.
We make use of 'Extract Groups' and 'Publishing Groups' to process batches of data
1
Creating ‘Extract Groups’
Extract Groups can be created in order to group the tables that are part of the Orchestration Workflow. Hence, if we want to include only 5 tables in our process, out of the 20 tables included in our connection to the data lake, we use Extract Groups.
Extract Groups are created in 'Modeler Connect' as shown below: we create an 'Extraction Group', name it, select the Data 'Connections' and then the individual tables within them.
2
Creating ‘Publishing Groups’
A similar concept is used for Publishing Groups. It helps us to select only those tables which we want to publish as part of 'Orchestration Workflow'.
Publishing Groups are created in 'Modeler Prepare' as shown below: we create a 'Publishing Group', name it and then select the Staging Tables to be put in.
3
Creating ‘Orchestration Workflow’
We create an 'Orchestration Workflow', which can be run manually or automatically. For our business problem, we will run the workflow automatically by scheduling it.
While configuring 'Orchestration Workflow', we imagine the steps that we usually do when updating the data. First we create a Workflow and name it.
We add steps for extracting the data from the data lake using 'Extract Groups', and then publishing the data using 'Publishing Groups' or all the data tables.
4
Scheduling ‘Orchestration Workflow’
Finally, we schedule the workflow as per requirements. An important factor to consider is to allow sufficient time window for the workflow to complete before the user checks the new data. The example below shows that the workflow will run on 2nd day of every month at 12:00 AM EST.
Several schedules can be added as per requirements.
Resources
On this page
- Keep analytics data updated