BizTalk Adapter for Google BigQuery

Build 20.0.7654

Connection Properties

Connection String Options  

The connection string properties describe the various options that can be used to establish a connection.

Adapter

CData.BizTalk.GoogleBigQueryAdapter.dll

Remarks

The connection string can be set to a series of "option=value" pairs, separated by semicolons. If a connection string property value has special characters such as semicolons, single quotes, spaces, etc., then you must quote the value using either single or double quotes.

Connection options are case insensitive.

To specify a location to the database where the tables, views, and stored procedures are located, set the Location property.

Authenticate via OAuth Authentication

Use the OAuth authentication standard to connect to Google BigQuery. You can authenticate with a user account or with a service account. A service account is required to grant organization-wide access scopes to the adapter. The adapter facilitates these authentication flows as described below.

Authenticate with a User Account

You can connect without setting any connection properties for your user credentials. After setting the following, you are ready to connect:

  • DatasetId: Set this to the Id of the dataset you want to connect to.
  • ProjectId: Set this to the Id of the project you want to connect to.
  • InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
When you connect the adapter opens the OAuth endpoint in your default browser. Log in and grant permissions to the application. The adapter then completes the OAuth process.
  1. Extracts the access token from the callback URL and authenticates requests.
  2. Obtains a new access token when the old one expires.
  3. Saves OAuth values in OAuthSettingsLocation to be persisted across connections.

See Using OAuth Authentication for other OAuth authentication flows.

Authenticate with a Service Account

Service accounts have silent authentication, without user authentication in the browser. You can also use a service account to delegate enterprise-wide access scopes to the adapter.

You need to create an OAuth application in this flow. See Creating a Custom OAuth App in the Getting Started section to create and authorize an app. You can then connect to Google BigQuery data that the service account has permission to access.

After setting the following connection properties, you are ready to connect:

  • InitiateOAuth: Set this to GETANDREFRESH.
  • OAuthJWTCertType: Set this to "PFXFILE".
  • OAuthJWTCert: Set this to the path to the .p12 file you generated.
  • OAuthJWTCertPassword: Set this to the password of the .p12 file.
  • OAuthJWTCertSubject: Set this to "*" to pick the first certificate in the certificate store.
  • OAuthJWTSubject: Set this to the email address of the user for whom the application is requesting delegate access. Note that delegate access must be granted by an administrator.
  • OAuthJWTIssuer: In the service accounts section, click Manage Service Accounts and set this field to the email address displayed in the service account Id field.
  • DatasetId: Set this to the Id of the dataset you want to connect to.
  • ProjectId: Set this to the Id of the project you want to connect to.
When you connect the adapter completes the OAuth flow for a service account.
  1. Creates and signs the JWT with the claim set required by the adapter.
  2. Exchanges the JWT for the access token.
  3. Saves OAuth values in OAuthSettingsLocation to be persisted across connections.
  4. Submits the JWT for a new access token when the token expires.

Connection String Options


The following is the full list of the options you can configure in the connection string for this provider. Click on the links for further details.

Allow Aggregate ParametersAllows raw aggregates to be used in parameters when QueryPassthrough is enabled.
Allow Large Result SetsWhether or not to allow large datasets to be stored in temporary tables for large datasets.
Allow Updates Without KeyWhether or not to allow update without primary keys.
Audit LimitThe maximum number of rows which will be stored within an audit table.
Audit ModeWhat provider actions should be recorded to audit tables.
Auth SchemeThe type of authentication to use when connecting to Google BigQuery.
Auto CacheAutomatically caches the results of SELECT queries into a cache database specified by either CacheLocation or both of CacheConnection and CacheProvider .
Big Query OptionsA comma separated list of Google BigQuery options.
Cache ConnectionThe connection string for the cache database. This property is always used in conjunction with CacheProvider . Setting both properties will override the value set for CacheLocation for caching data.
Cache LocationSpecifies the path to the cache when caching to a file.
Cache MetadataThis property determines whether or not to cache the table metadata to a file store.
Cache ProviderThe name of the provider to be used to cache data.
Cache ToleranceThe tolerance for stale data in the cache specified in seconds when using AutoCache .
Dataset IdThe DatasetId of the dataset you wish to connect to and view tables of.
Destination TableDestinationTable is entered in the format DestinationProjectId:DestinationDataSet.TableName. Setting DestinationTable allows the query to return very large result sets (more than 128mb of result data) to a permanent destination table.
Filter ColumnsPlease set `AllowUpdatesWithoutKey` to true before you could use this property.
Firewall PasswordA password used to authenticate to a proxy-based firewall.
Firewall PortThe TCP port for a proxy-based firewall.
Firewall ServerThe name or IP address of a proxy-based firewall.
Firewall TypeThe protocol used by a proxy-based firewall.
Firewall UserThe user name to use to authenticate with a proxy-based firewall.
Generate Schema FilesIndicates the user preference as to when schemas should be generated and saved.
Initiate OAuthSet this property to initiate the process to obtain or refresh the OAuth access token when you connect.
LocationA path to the directory that contains the schema files defining tables, views, and stored procedures.
LogfileA filepath which designates the name and location of the log file.
Log ModulesCore modules to be included in the log file.
Maximum Billing TierThe MaximumBillingTier is a positive integer that serves as a multiplier of the basic price per TB. For example, if you set MaximumBillingTier to 2, the maximum cost for that query will be 2x basic price per TB.
Maximum Bytes BilledLimits how many bytes BigQuery will allow a job to consume before it is cancelled.
Max Log File CountA string specifying the maximum file count of log files. When the limit is hit, a new log is created in the same folder with the date and time appended to the end and the oldest log file will be deleted.
Max Log File SizeA string specifying the maximum size in bytes for a log file (for example, 10 MB). When the limit is hit, a new log is created in the same folder with the date and time appended to the end.
Max RowsLimits the number of rows returned rows when no aggregation or group by is used in the query. This helps avoid performance issues at design time.
OAuth Access TokenThe access token for connecting using OAuth.
OAuth Client IdThe client ID assigned when you register your application with an OAuth authorization server.
OAuth Client SecretThe client secret assigned when you register your application with an OAuth authorization server.
OAuth Expires InThe lifetime in seconds of the OAuth AccessToken.
OAuth JWT CertThe JWT Certificate store.
OAuth JWT Cert PasswordThe password for the OAuth JWT certificate.
OAuth JWT Cert SubjectThe subject of the OAuth JWT certificate.
OAuth JWT Cert TypeThe type of key store containing the JWT Certificate.
OAuth JWT IssuerThe issuer of the Java Web Token.
OAuth JWT SubjectThe user subject for which the application is requesting delegated access.
OAuth Refresh TokenThe OAuth refresh token for the corresponding OAuth access token.
OAuth Settings LocationThe location of the settings file where OAuth values are saved when InitiateOAuth is set to GETANDREFRESH or REFRESH. Alternatively, this can be held in memory by specifying a value starting with memory://.
OAuth Token TimestampThe Unix epoch timestamp in milliseconds when the current Access Token was created.
OAuth VerifierThe verifier code returned from the OAuth authorization URL.
OfflineUse offline mode to get the data from the cache instead of the live source.
OtherThese hidden properties are used only in specific use cases.
Page SizeThe number of results to return per page from Google BigQuery.
Polling IntervalThis determines how long to wait in seconds, between checks to see if a job has completed.
Project IdThe ProjectId of the billing project for executing jobs.
Proxy Auth SchemeThe authentication type to use to authenticate to the ProxyServer proxy.
Proxy Auto DetectThis indicates whether to use the system proxy settings or not. This takes precedence over other proxy settings, so you'll need to set ProxyAutoDetect to FALSE in order use custom proxy settings.
Proxy ExceptionsA semicolon separated list of destination hostnames or IPs that are exempt from connecting through the ProxyServer .
Proxy PasswordA password to be used to authenticate to the ProxyServer proxy.
Proxy PortThe TCP port the ProxyServer proxy is running on.
Proxy ServerThe hostname or IP address of a proxy to route HTTP traffic through.
Proxy SSL TypeThe SSL type to use when connecting to the ProxyServer proxy.
Proxy UserA user name to be used to authenticate to the ProxyServer proxy.
Pseudo ColumnsThis property indicates whether or not to include pseudo columns as columns to the table.
Query PassthroughThis option passes the query to the Google BigQuery server as is.
ReadonlyYou can use this property to enforce read-only access to Google BigQuery from the provider.
Refresh View SchemasAllows the provider to determine up-to-date view schemas automatically.
RTKThe runtime key used for licensing.
ScopeIdentifies the Google API access that your application is requesting. Scopes are space-delimited.
Show Table DescriptionsControls whether table descriptions are returned via the platform metadata APIs and sys_tables / sys_views.
SSL Server CertThe certificate to be accepted from the server when connecting using TLS/SSL.
Storage Page SizeSpecifies the page size to use for Storage API queries.
Storage ThresholdThe minimum number of rows a query must return to invoke the Storage API.
TablesThis property restricts the tables reported to a subset of the available tables. For example, Tables=TableA,TableB,TableC.
Temp Table DatasetThe prefix of the dataset that will contain temporary tables when executing queries with large result sets.
Temp Table Expiration TimeTime, in seconds until the temporary table expires. Set to 0 to have the table never expire.
TimeoutThe value in seconds until the timeout error is thrown, canceling the operation.
Use Arrow FormatSpecifies whether to use the Arrow format with BigQuery's Storage API.
Use Legacy SQLSpecifies whether to use BigQuery's legacy SQL dialect for this query. By default, Standard SQL will be used.
Use Query CacheSpecifies whether to use Google BigQuery's built-in query cache.
Use Storage APISpecifies whether to use BigQuery's Storage API for bulk data reads.
Use Streaming InsertsSpecifies whether to use streaming or DML statements when inserting data. By default, streaming inserts are used.
ViewsRestricts the views reported to a subset of the available tables. For example, Views=ViewA,ViewB,ViewC.

Copyright (c) 2020 CData Software, Inc. - All rights reserved.
Build 20.0.7654