SAP Cloud Platform Open Connectors

Introducing Bulk v3

The new bulk v3 framework provided by SAP Cloud Platform Open Connectors platform includes several APIs to help streamline the creation and management of your bulk jobs. This article describes the available bulk APIs and the job criteria parameters. In addition to the information in this article, you can also find the bulk APIs in a Postman collection here.

Bulk v3 APIs

The bulk v3 framework APIs are described in the following table:

APIDescription
POST /bulk/download
Initiates a bulk job to download all the records as per the given job criteria, which is specified as part of the request body. For further information on job-criteria, see the Job Criteria section.
GET /bulk/:id/status

Provides the status of the job. The response will include the following information in a JSON format:

{
    "id": Double,
    "recordCount": Double,  //Number of records downloaded.
    "objectName": "string",
    "metaData": JSONObject, //This will mostly contian user inputs form /bulk/download API and some custom API specific data as per element.
    "jobDirection": "string", //DOWNLOAD or UPLOAD
    "jobStatus": "string",  //This will be a chronical status of the job.  CREATEd, STARTED, RUNNING, ABORTED, COMPLETED, CANCELED, CANCELLATION_PENDING
 
    "statusMessage": "stirng",  //This is a more comprehensive state of the job designed for user convenience.  For example: `Successfully downloaded 100 records`
    "createdDate": 1558398989506,  //Job created date.
    "vendorJobId": "string",  //If vendor supports native bulk/batch proceessing, we can witness this.
    "errorCount": Double,
    "notificationUrl": "string",
    "defaultSelectFields": "string",
    "completionTime": Date, //Job completion time.
    "fileFormat": "string", //Can be JSON, CSV, or JSONL.
    "jobCriteria": JSONObject,  //This is the request body from /bulk/download
    "native": Boolean //If the vendor supports native bulk jobs and if we are running the job through vendor this will be true otherwise false
}
GET /bulk/:idProduces the results or data of the download job in the form of a streaming file. The default format of the file will match the specified format in the Job-Criteria; if no format is defined, the default value is JSONL format.

GET /bulk/:id/data:

Paginates the results in the bulk file that we can download through the /bulk/:id API.
PUT /bulk/:id/cancel
Cancels the job at any time, provided the job is not in COMPLETED or ABORTED state.
GET /bulk/:id/errors
Yields the errors that occurred during the job runtime.
PUT /bulk/v3/{id}/restart

Continues a job where it left off, this scenario may occur if there is an outage in the downstream service, API limits are hit on the vendor side, or the apiLimit set on the bulk request was reached for that day. 

Job Criteria

The following table lists and describes the configuration parameters that can be sent as a payload to the /bulk/download API.

ParameterDescription
objectName*
The resource name for which you want to download the records. This should match with the API supported in the connector. For example, a Salesforce Sales Cloud user wants all the accounts available in his/her account. In this case, the objectName will be accounts.
format*
Allows the user to choose the format that a user wishes to download. The supported values for this filed are application/json, text/csv, and application/jsonl.
selectFields
Gives the user the ability to retrieve whichever fields they are interested in. For example, the accounts resource may support over 100 fields, but the user is interested in fetching only a few fields like accountName, LasModifiedDate, and id.  This use case can be solved through selectFields by passing the field value like accountName,LasModifiedDate,id as a comma-separated list of field names.
limit
Restricts the bulk job to download a certain amount of records rather downloading all records. When the job reaches a number of records equivalent to the specified limit, the job automatically stops and changes its state to COMPLETED.
apiLimit
Throttles the bulk job when the API call count reaches a specified value. This field is helpful when service providers offer only a certain amount of calls per job.
filterNulls
Removes null values from the response payload when specified true. Otherwise, no filtering will happen on the response payload for nulls.
from
Accepts a date as a value in ISO 8601 Date format, which is helpful to filter the records from the given date against the date field in the response payload, which can be identified through the filterDateField. In other words, the bulk job will skip the records which are identified to be modified or inserted before the date given with from. Note that if from date is given, filterDateField should also be present in the request body.
to
Works the other way around from field, meaning the bulk job will skip all the records which are identified to be modified or inserted after the date given with from. Note that if to date is given, filterDateField should be present in the request body.
filterDateField
Required when from or to dates are given, a conditionally required field with from and to. This filed helps the bulk job identify which field to compare with when from or to date is given.
notificationUrl
Accepts the notification URL at which the user will be notified when the bulk job is complete.
where
A well-known typical OCNQL query for any SEARCH API. This can help improve the performance of the job and directly apply the filtering to the service provider.
query
Any other API related and supported filters (query params) will be accepted through this field, which is of type JSONObject (Map).
pageSize
The default value is 200 records.
docsHubDetails

If a user wants to upload their file to a documentation hub once the job completed, we use this field to automatically collect some information about the documentation hub and streamline the process.

This feature is supported through our documentation hub connectors, eg Google Drive, Dropbox, Box, etc. This field accepts a JSON map as its value with the following properties:

{
    "instanceId": "string",  //instance id is the id of one of the documentation hub element
    "path": "string"        //speicified the location, where to upload the file
}


continueFromJobId

This flag helps to initiate a bulk download job with respect to another job that was completed successfully.  This property accepts another bulk job id as the value and when submitted, the new job considers the context of the old job (that will be identified with the given id) and starts downlaod data from then.