Search

Data Migration from on-premise / hybrid cloud

Overview

Data silos scattered across the enterprise make it cost prohibitive and technically challenging to harness its value through business intelligence or machine learning.

This is where the Infor Data Lake, an easily scalable and comprehensive central repository for the enterprise data, can help solve these challenges.

Components

Requirements

Tutorial

Difficulty: Medium
Estimated Completion Time:60 minutes

In this tutorial, we walk through how you can easily migrate on-premise or private cloud data into the Infor Data Lake. In this example, we will be migrating tables from AdventureWorks, an OLTP sample database of a fictitious bicycle parts wholesaler with 300 employees, 500 products, 20,000 customers and 31,000 sales.

1. Download and install the PostgreSQL client.

Note that Postgres is used as an example in this tutorial, you are able to connect to other database systems.

2. Follow these steps to create a local database.

3. Follow these steps to load the sample data into your local database.

4. Install and configure the Enterprise Connector (detailed instructions)

5. Enable Data Loader under the ION Services Preview Features.

6. Create an authorized app in ION API gateway and download the credentials (detailed instructions)

7. Follow the instructions in the video below to set up the Data Loader

The video goes through how to pull data from the on-premise database, through the enterprise connector, and into the data lake.

Full tutorial on how to use Data Loader and Data Catalog Crawler

Best Practices

When registering objects in Data Catalog using the Metadata Crawler, it is recommended to properly and diligently define the object properties, these include:

  • Identifier Paths: The JSON paths to the properties that uniquely identify the object.
  • Variation Path: The JSON path to the property that contains the variation value for the object.
  • Delete Indicator: The JSON path to the property that is used to indicate whether the object has been marked as deleted.
  • Delete Indicator Value: The value that the delete indicator must have to indicate that the object has been marked as deleted.
  • Timestamp Path: The JSON path to the property that contains the timestamp of the moment the object was created or updated.
  • Archive Indicator: The JSON path to the property that is used to indicate whether the object has been archived in the source application.
  • Archive Indicator Value: The value that the archive indicator must have to indicate that the object has been archived.

Specifically, for efficient variation handling in Data Lake, the Identifier Paths and Variation Path must be defined and leveraged properly.