Data Migration from on-premise / hybrid cloud
Business Problem
Data silos scattered across your enterprise make it cost prohibitive and technically challenging to harness its value through business intelligence or machine learning.
This is where the Infor Data Lake, an easily scalable and comprehensive central repository for your enterprise data, can help solve these challenges.
Components
Requirements
- Sample Database: AdventureWorks
- Access to an Infor CloudSuite
- User privileges to ION Desk (IONDeskAdmin), Data Fabric (DATAFABRIC-SuperAdmin), and ION API Gateway (IONAPI-Administrator)
- Access to install the Enterprise Connector
- PostgreSQL server installed locally and SQL knowledge to load sample data
- Optional Campus classes:
- Infor OS: Foundation for Multi-Tenant – Part 1
- Infor OS: Foundation for Multi-Tenant – Part 2
Tutorial
In this tutorial, we walk through how you can easily migrate on-premise or private cloud data into the Infor Data Lake. In this example, we will be migrating tables from AdventureWorks, an OLTP sample database of a fictitious bicycle parts wholesaler with 300 employees, 500 products, 20,000 customers and 31,000 sales.
- Download and install the PostgreSQL client. Note that Postgres is used as an example in this tutorial, you are able to connect to other database systems.
- Follow these steps to create a local database.
- Follow these steps to load the sample data into your local database.
- Install and configure the Enterprise Connector (detailed instructions)
- Enable Data Loader under the ION Services Preview Features
- Create an authorized app in ION API gateway and download the credentials (detailed instructions)
- Follow the instructions in the video below to setup the Data Loader to pull data from the on-premise database, through the enterprise connector, and into the data lake.
Best Practices
When registering objects in Data Catalog using the Metadata Crawler, it is recommended to properly and diligently define the object properties, these include:
- Identifier Paths: The JSON paths to the properties that uniquely identify the object.
- Variation Path: The JSON path to the property that contains the variation value for the object.
- Delete Indicator: The JSON path to the property that is used to indicate whether the object has been marked as deleted.
- Delete Indicator Value: The value that the delete indicator must have to indicate that the object has been marked as deleted.
- Timestamp Path: The JSON path to the property that contains the timestamp of the moment the object was created or updated.
- Archive Indicator: The JSON path to the property that is used to indicate whether the object has been archived in the source application.
- Archive Indicator Value: The value that the archive indicator must have to indicate that the object has been archived.
Specifically, for efficient variation handling in Data Lake, the Identifier Paths and Variation Path must be defined and leveraged properly.