infor

Manage

This API is used to manage the value of `dl_corrupt` property for a data object.
For a data object, using the dl_id property, you can set its dl_corrupt property to either TRUE or FALSE.
By marking a data object as *corrupted* (`dl_corrupt = true`) a client specifies that the data object content contains invalid data. Data objects which are marked as corrupt, are omitted by:
   - /byfilter API, which by default ignores the data objects set to be corrupted.
   - Scanning by analysis processes (e.g. import to Data Lake Services).


Change the corrupt property of a data object.

This API displays a list of log records containing details about performed purge actions.

The list contains records that were initiated from the Data Lake Purge API or from ION Desk - Data Lake Purge page.

It returns an array of purge log objects. Each object contains following attributes:

    purgeOperationId: purge event identifier,
    purgeOperationStart: date representation when the purge process started,
    purgeOperationEnd: date representation when the purge process ended,
    filter: filter by which objects were selected from datalake to be purged,
    purgedBy: name of the user that initiated the purge process,
    totalCountToPurge: number of the objects selected by the filter that were going to be purged,
    totalSizeToPurge: size of the objects selected by the filter that were going to be purged,
    purgedCount: number of the objects that were purged during the process,
    purgedSize: size of the objects that were  purged during the process,
    source: source, where the purge request originates from (ION Desk or API),
    status: defines the state of the purge process (RUNNING, SUCCESS, ERROR, CANCELLED),
    errorCode: code of the error if error occurs,
    errorMessage: textual explanation of the cause of the error if error occurs,
    cancelledBy: name of the user that cancelled the running purge process if the process was cancelled,




Method lists purge log records.

The API is used to delete data objects from the Data Lake based on their unique identifier: `dl_id`.
The dl_id is retrievable by using the Data Lake Payloads API.
Object depicting the result of the purge operation containing following fields:

    message: Human readable status of the purge operation
    usedFilter: Filter utilized to purge the DataLake objects
    purgedAllByFilter:
        TRUE = current purge operation has deleted all DataLake objects that conform the given filter
        FALSE = current purge operation has deleted specified number of DataLake objects given by the filter, but the operation is needed to be repeated because there are still some objects left in the DataLake
    purgedNumber: Number of objects purged within this purge process


Method deletes Data Objects based on the given Data Object identifiers.

This API is used to delete objects from the DataLake based on the query filter parameter.
The number of objects that could be deleted in a single batch size is limited to 1000. The same API call can be repeated in case there exist more than 1000 data objects to be purged, with the same filter.
When the response body contains the property 'purgedAllByFilter', with a value TRUE, it means all data objects matching the filter parameter query were deleted. 
Object depicting the result of the purge operation containing following fields:

    message: Human readable status of the purge operation
    usedFilter: Filter utilized to purge the DataLake objects
    purgedAllByFilter:
        TRUE = current purge operation has deleted all DataLake objects that conform the given filter
        FALSE = current purge operation has deleted specified number of DataLake objects given by the filter, but the operation is needed to be repeated because there are still some objects left in the DataLake
    purgedNumber: Number of objects purged within this purge process


*The use of the filter parameter is identical as documented in the `/dataobjects` API method.*


Method deletes Data Objects based on the given filter.

Statistics about data stored in Data Lake

Stats

Description:

   This API is used to get Aggregation of the objects in Infor Data Lake organized by TenantId

Input:

   We need specify a time range for a query we have two options for it

       first:
           "startDate" - "2018-06-25T11:55:14.970Z" define start the time range
           "endDate" - "2018-06-25T11:55:15.646Z" define end the time range
               "2018" - year 1000 - 9999
               "06" - a month 01 - 12
               "25" - a day 01 - [28,29,30,31]
               "11" - an hour 00 - 23
               "55" - a minute 00 - 59
               "15" - a second 00 - 59
               "646" - a millisecond 000 - 999
       second:
           "timeRange" - define a time range for Query like last hour (now-timeRange - now)
   Only one option can use for a query at the same time "first" or "second"


   "step" - define a sub aggregation interval

       time range and step can be specified with follow parameters
           "1m" - 1 minute
           "1h" - 1 hour
           "1d" - 1 day
           other options:
               "y" - years
               "M" - months
               "w" - weeks
               "d" - days
               "h/H" - hours
               "m" - minutes
               "s" - seconds
       please note fractional time values are not supported 1.5h could instead be specified as 90m
   "sourceApplication" - String "IOT" or "ION" if this field doesn't specified this filter will not be use

   "attributes" - this is an array of attributes each of them specified an aggregation

       "property" - String datalake field name for aggregation
           following fields available  for aggregation: 
               "dl_id", "dl_instance_count" and "dl_size"
       "aggregationType" - String aggregation type can be one of "sum", "count", "min", "max", "avg"
           available methods depends on the field:
               "dl_id" - "count"
               "dl_instance_count" - "sum", "count", "min", "max', "avg"
               "dl_size" - "sum", "count", "min", "max', "avg"


Example call:

   POST /datalake/v2/stats/aggregation

BODY of the POST request:

      {
           "timeRange": "1d",
           "step" : "1h",
           "sourceApplication" : "IOT",
           "attributes" : [
               {
                   "property" : "dl_instance_count",
                   "aggregationType" : "count"
               }
             ]
      }


The service will return the following information:

Output:

   Json representation of DLAggregationModel

      {
           "timeRange": "1d", - query time range
           "step": "1h", - sub aggregation step
           "dataForCustomPeriod": {
               "2018-06-15T08:00:00.000Z": {
                   "data": [
                       {
                           "property": "dl_instance_count", - datalake aggregation field name
                           "aggregationType": "count", - type of aggregation
                           "resultValue": 8 - calculation result
                       }
                   ]
               }
           }
       }


Aggregate summary user defined field on tenant from datalake.

Description:

   This API is used to get the top 10 biggest document names stored by their size grouped by Data object names

   The service will return the following information:

       DLTopObjectsModel = {
           dataSinceBeginning: [map] - map where the key is the data type and the value is a list of TopObject,
           dataLastWeek: [map] - map where the key is the the data type and the value is a list of TopObject - filtered by last week
           dataLastDay: [map of OverviewObject] - map where the key is the the data type and the value is a list of TopObject - filtered by last day
       }

   Where TopObject = {
           name: [String] - document name
           value: [long] - size
         }



Output:
   Json representation of DLTopObjectsModel



Example call:

   GET /datalake/v2/stats/topObjects





The top 10 biggest Data Objects Names stored by their size.

Description:

   This API is used to get an overview of the Infor Data Lake

   The service will return the following information:

       DLOverviewModel = {
           size: [double] - sum of the Infor Data Lake object sizes in bytes,
           objectCount: [long] - number of data objects in the Infor Data Lake
           data: [array of OverviewObject] - number of messages per type in the Infor Data Lake
       }

   Where OverviewObject = {
           name: [String]
           value: [long]
         }



Output:
   Json representation of DLOverviewModel



Example call:

   GET /datalake/v2/stats/overview



Example output:

   {
       "size":712565155",
       "objectCount":42575"
       "data":[{"name":"bod", "value" : 52567028}]"
   }




Get the total number of data objects and total size.

Description:

   This API is used to get size and count of the objects in Infor Data Lake organized by time

   The service will return the following information:

       DLDataIngestionModel = {
           dataSinceBeginning: [map] - map where the key is the time of the month and the value is a Stat object,
           dataLastWeek: [map] - map where the key is the time of the day and the value is a Stat object - filtered by last week
           dataLastDay: [map] - map where the key is the time of the hour and the value is a Stat object - filtered by last day
       }

   Where Stat = {
           count: [long] - number of objects
           size: [long] - overall size
         }



Output:
   Json representation of DLDataIngestionModel



Example call:

   GET /datalake/v2/stats/dataingestion





Size and count of the documents in Infor Data Lake

General

Get the current running build number of the Infor Data Lake

Ping call for the Infor Data Lake api

Retrieve data objects stored in Data Lake

Retrieve

This API is used to list the properties of data objects that are stored in Data Lake or have been archived.
'filter' URL parameter can be used to express "dynamic" search query by the client. It is composed of conditions, which are connected with boolean operators.

*Simple condition:*
   - `(field_name operator 'value')`

*Conditions can be connected with "and" and "or" operators:*
   - `(dl_document_name eq '*SalesOrder') and (dl_from_logical_id eq 'infor.jdbc.test_cp') and dl_document_date range [2017-12-19T23:24:22.410Z, 2017-12-19T23:24:22.700Z]`

*Conditions can be nested as follows:*
   - `(dl_document_name eq '*jdbc') and (dl_from_logical_id eq 'infor.jdbc.test_cp' OR (dl_from_logical_id eq 'infor.jdbc.test_cp' AND dl_header_accounting_entity eq "test1 c") )`

---
**Condition structure breakdown:**
   - *field_name:* name of field that is going to be compared
       - A name of one of the attributes, belonging to the queried collection
   - *operator:* one of the binary operators for operators list below
   - *value:* Value to compare to.
       - If value is string, it is enclosed in single or double quotes. It is case sensitive.
           - double quote: value is phrase(the whole phrase is matched)
           - single quote: value is word. It can contain wild characters: *
       - If value is a number, it is not enclosed
       - If value is date:
           - If condition is 'range' : date is not enclosed and conforms to DATE_ISO8601_FORMAT = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"; e.g. 2017-12-19T23:24:22.410Z
           - If condition is 'eq' : date needs to be enclosed in single or double quotes (wildcard is NOT applied)

---
**Operator list:**
   - *eq* - equals
   - *gt* - greater than
   - *ge* - greater than or equal
   - *lt* - less than
   - *le* - less than or equal
   - *range* - range of dates

**Logical operator list:**
   - *and* - and
   - *or* - or

**Values:**
   - Numeric values are not enclosed in quotes
   - Decimal numbers - dot character is used as a separator for numeric values with decimal part
       - *Example*: 123.45

**Ranges:**
   - *[value1, value2]* - square brackets defines inclusion of the value
      - *Example*: [1, 5] - range of values 1, 2, 3, 4, 5; both start and end values are included in the range
   - *{value1, value2}* - curly brackets defines exclusion of the value
      - *Example*: {1, 5} - range of values 2, 3, 4; both start and end values are excluded from the range

---
**Value examples:**
   - `(dl_from_logical_id eq 'infor.jdbc.test_cp')` // String, enclosed in single. Returns records where custom contains specified word. It accepts wildcard *.
   - `(dl_from_logical_id eq "infor.jdbc.test_cp")` // String, enclosed in double quotes is phrase. It returns the exact match.
   - `(dl_from_logical_id eq "\*test\*")` // String, enclosed in double quotes cannot contain wildcards
   - `(size lt 35)` // Number, not enclosed
   - `dl_document_date range [2017-12-19T23:24:22.410Z, 2017-12-19T23:24:22.700Z]` // both start and end timestamp are included in the result
   - `dl_document_date range [2017-12-19T23:24:22.410Z, 2017-12-19T23:24:22.700Z}` // only start timestamp is included in the result; objects having the date equal to the end timestamp are excluded
   - `dl_document_date eq "2017-12-19T23:24:22.410Z"` // Exact match date

---
**Queryable data object properties:**
   - `dl_id` - Identifier of the data object residing in the Data Lake.
   - `event_date` - Date time in which the event associated to the data object was generated. Format: `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'`.
   - `dl_document_date` - Date time in which the data object was ingested into the Data Lake. Format: `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'`
   - `dl_document_indexed_date` - Date time when the object became visible in Data Lake after it was stored. `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'`
   - `dl_instance_count` - Number of instances in the data object.
   - `dl_message_id` - Identifier of the message that transported the data object.
   - `dl_corrupt` - A marker for corrupted data objects as determined by the Data Lake MarkCorrupt API.
   - `dl_compression_type` - Indicates the compression type applied to the data object. DEFLATE is default.
   - `dl_encoding` - Indicates the type of encoding on the raw object.
   - `dl_document_name` - Indicates the document name of the data object with direct correlation to the document name in the Infor Data Catalog.
   - `dl_size` - Data object size in bytes.
   - `dl_from_logical_id` - Identifier of the provider of the data object.
   - `dl_header_accounting_entity` - The Accounting entity of the business object that is in the data object.
   - `dl_header_location` - The location of the business object that is in the data object.
   - `dl_header_document_id` - Document ID is the simple ID for this object in its context, for example respective accounting entity, tenant etc.
   - `dl_header_source` - Source header can be used to preserve information about message source, for example file name in case of File Connector.
   - `dl_revision_id` - Revision ID sets the value that is used to keep multiple versions of the same document unique.
   - `dl_variation_id` - The VariationID can be used at the receiving end to discover messages that are received out of sequence.
   - `dl_batch_id` - The unique ID of the batch. (ION-specific property)
   - `dl_batch_sequence` - The sequence number of this BOD in the batch. (ION-specific property)
   - `dl_batch_size` - The total number of BODs in the batch. This can be unknown until the last BOD. (ION-specific property)
   - `dl_batch_revision` - The revision number of the batch. This is used when one set of BODs fails. The complete set can be resent using a new revision number. (ION-specific property)
   - `dl_batch_abort_indicator` - This indicator type attribute is set to true when a sending system determines that it will not finish sending a batch. The receiving system is notified that any messages received as a part of this batch must be discarded. (ION-specific property)
   - `dl_source_publication_date` - Date and time representing the time at which a data object was published by a source application. Value should be sent as DateTime (String) - UTC ISO 8601 "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
   - `time_in_transit` - The time in which a data object spent transiting from a source application to receipt in Data Lake
   - `dl_channel` - The means by which data was ingested into the Data Fabric.
   - `dl_archived` - Archived data objects possess archived records that may no longer exist within the system of record.



List data object properties using a filter.

Method provides the content of the Data Object based on the provided `dl_id` property.

Client can specify the value of the 'Accept-encoding' header to set the content encoding (compression).
**Accepted values are:**

   - *deflate:* to return the compressed content (DEFLATE algorithm)
   - *identity:* to return the non-compressed content



Retrieve payload based on id from datalake

If the filter query (the one provided as the method parameter) produces more than 10K results, it is impossible to display all the results. The filter query must be refined to respond with less than 10K results.
The purpose of this API is to allow a client to retrieve such a large amount of results when needed.
The response of this API is a set of filters, that when used (e.g. /byfilter API) will ensure less than 10K results in the response.Combining the split filters from the response and using them in sequence, will provide the same result as the original (large) filter query.
In case, that the original filter response is less than 10K objects, the API response with the original query.

*The use of the filter parameter is identical as documented in the `/dataobjects` API method.*
V2 allows clients to specify the size of the results produced from the split filter. 

Split a demanding filter (producing more than 10K results) into several smaller filters producing the same final result (up to 9500 results per one filter).

This API method returns the content of multiple data objects in the *multipart* form.
Each multipart object body is composed of two parts:
- Data Object properties
- Data Object content

Client can specify the value of the 'Accept-encoding' header to set the content encoding (compression).
**Accepted values are:**

   - *deflate:* to return the compressed content (DEFLATE algorithm)
   - *identity:* to return the non-compressed content



*The use of the filter parameter is identical as documented in the `/dataobjects` API method.*


Sections