Customizing IBM Cloud Data Engine Behavior

Follow the steps below to configure the connector defaults for querying IBM Cloud Data Engine.

Checking the Job Status

Once a job is initiated server-side, IBM Cloud Data Engine automatically checks for this job to finish processing. You can use PollingInterval to configure the wait time of the requests that the provider is doing towards the job status.

Note that some queries may be complex and it will take longer for the result to be retrieved, that is why the PollingInterval might be configured to be longer in order not to make so many unnecessary requests. A job is initiated server-side only when the SELECT query is not a simple "Select * FROM Table" query but it has at least a criteria or projection in it. The outcome of each job is considered a view as it supports only "Select" operation.

Cleaning Query Results

IBM Cloud Data Engine stores the results of every query you execute in CSV or JSON files in ResultLocationURI. You can use CleanQueryResults, to automatically clean these files for every query executed.

Note that this behavior will add a minor performance hit when you disconnect the last connection in a process.

Using IBM Cloud's Query Caching

You configure QueryCachingLevel to modify the usage of the query results stored in ResultLocationURI. This is especially helpful when executing a certain query multiple times. This means IBM Cloud Data Engine will not scan the same data again and simply use the results from the previous execution. These results are not used if the job was started earlier than the amount of seconds specified in QueryTolerance.

Default Bucket

A default bucket may be specified by setting the Bucket and connection property. This is optional and will be used if you do not specify a bucket when issuing a query. If no Bucket is set, then the bucket name must be used as the schema in the query in the format [Schema].[Table].

Storing Results

Results may be stored in different data types. By default, the CData Python Connector for IBM Cloud Data Engine will use CSV. This may be altered by modifying StoreResultsAs.