Connecting to HTTP Streams

Authenticating to HTTP(S)

The connector generically supports connecting to Avro data stored on HTTP(S) streams.

Several authentication methods, such as user/password, digest access, OAuth, OAuthJWT, and OAuth PASSWORD flow are supported.

You can also connect to streams that have no authentication set up.

No Authentication

Connect to an HTTP(S) stream with no authentication by setting the AuthScheme connection property to None.

Basic

Set the following to connect:

AuthScheme: Set this to Basic.
User: Set this to the username associated with your HTTP(S) stream.
Password: Set this to the password associated with your HTTP(S) stream.

Digest

Set the following to connect:

AuthScheme: Set this to Digest.
User: Set this to the username associated with your HTTP(S) stream.
Password: Set this to the password associated with your HTTP(S) stream.

OAuth

Set the AuthScheme to OAuth.

OAuth requires the authenticating user to interact with Avro using the browser. The connector facilitates this in various ways as described in the following sections.

Before following the procedures below, you need to register an OAuth app with the service containing the Avro data you want to work with.

Creating a custom application in most services requires registering as a developer and creating an app in the UI of the service.

This is not necessarily true for all services. In some you must contact the service provider to create the app for you. However it is done, you must obtain the values for OAuthClientId, OAuthClientSecret, and CallbackURL.

Desktop Applications

After setting the following connection properties, you are ready to connect:

OAuthVersion: Set this to the OAuth Version, either 1.0 or 2.0.
OAuthRequestTokenURL: Required for OAuth 1.0. In OAuth 1.0, this is the URL where the app makes a request for the request token.
OAuthAuthorizationURL: Required for OAuth 1.0 and 2.0. This is the URL where the user logs into the service and grants permissions to the application. In OAuth 1.0, if permissions are granted, the request token is authorized.
OAuthAccessTokenURL: Required for OAuth 1.0 and 2.0. This is the URL where the request for the access token is made. In OAuth 1.0, the authorized request token is exchanged for the access token.
OAuthRefreshTokenURL: Required for OAuth 2.0. In OAuth 2.0, this is the URL where the refresh token is exchanged for a new access token when the old one expires. Note that for your data source this may be the same as the access token URL.
OAuthClientId: Set this to the client Id in your app settings. This may also be called the consumer key.
OAuthClientSecret: Set this to the client secret in your app settings. This may also be called the consumer secret.
CallbackURL: Set this to http://localhost:33333. If you specified a redirect URL in your app settings, this must match.
InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the access token in the connection string.

When you connect, the connector opens the OAuth endpoint in your default browser. Log in and grant permissions to the application. The connector then completes the OAuth process:

Extracts the access token from the callback URL and authenticates requests.
Refreshes the access token when it expires.
Saves OAuth values in OAuthSettingsLocation to be persisted across connections.

Web Application

When connecting via a Web application, or if the connector is not authorized to open a browser window, use the provided stored procedures to get and manage the OAuth token values.

Set Up the OAuth Flow

Provide the OAuth URLs to authenticate in the Web flow.

OAuthRequestTokenURL: Required for OAuth 1.0. In OAuth 1.0, this is the URL where the app makes a request for the request token.
OAuthAuthorizationURL: Required for OAuth 1.0 and 2.0. This is the URL where the user logs into the service and grants permissions to the application. In OAuth 1.0, if permissions are granted, the request token is authorized.
OAuthAccessTokenURL: Required for OAuth 1.0 and 2.0. This is the URL where the request for the access token is made. In OAuth 1.0, the authorized request token is exchanged for the access token.
OAuthRefreshTokenURL: Required for OAuth 2.0. In OAuth 2.0, this is the URL where the refresh token is exchanged for a new access token when the old one expires. Note that for your data source this may be the same as the access token URL.

Get an Access Token

In addition to the OAuth URLs, set the following additional connection properties to obtain the OAuthAccessToken:

OAuthClientId: Set this to the client Id in your app settings. This may also be called the consumer key.
OAuthClientSecret: Set this to the client secret in your app settings. This may also be called the consumer secret.
OAuthVersion: Set this to the OAuth version, either 1.0 or 2.0.

You can then call stored procedures to complete the OAuth exchange:

Call the GetOAuthAuthorizationURL stored procedure. Set the AuthMode input to WEB and set the CallbackURL input to the Redirect URI you specified in your app settings. The stored procedure returns the URL to the OAuth endpoint.
Log in and authorize the application. You are redirected back to the callback URL.
Call the GetOAuthAccessToken stored procedure. Set the AuthMode input to WEB.

In OAuth 1.0, set the Verifier input to the "oauth_verifier" parameter. Extract the verifier code from the callback URL. Additionally, set the AuthToken and AuthSecret to the values returned by GetOAuthAccessToken.

In OAuth 2.0, set the Verifier input to the "code" parameter in the query string of the callback URL.

Connect to Data and Refresh the Token

The OAuthAccessToken returned by GetOAuthAccessToken has a limited lifetime. To automatically refresh the token, set the following on the first data connection. Alternatively, use the RefreshOAuthAccessToken stored procedure to manually refresh the token.

OAuth Endpoints

OAuthRequestTokenURL
OAuthAuthorizationURL
OAuthAccessTokenURL
OAuthRefreshTokenURL

OAuth Tokens and Keys

OAuthClientId
OAuthClientSecret
OAuthRefreshToken
OAuthAccessToken

Initiate OAuth

OAuthVersion: Set this to 1.0 or 2.0.
InitiateOAuth: Set this to REFRESH. You can use InitiateOAuth to avoid repeating the OAuth exchange and manually setting the OAuthAccessToken.
OAuthSettingsLocation: Set this to the location where the connector will save the OAuth values, to be persisted across connections.

On subsequent data connections, set the following:

InitiateOAuth
OAuthSettingsLocation
OAuthRequestTokenURL
OAuthAuthorizationURL
OAuthAccessTokenURL
OAuthRefreshTokenURL

OAuth JWT

Set AuthScheme to OAuthJWT.

The connector supports using JWT as an authorization grant in situations where a user cannot perform an interactive sign-on. After setting the following connection properties, you are ready to connect:

OAuthVersion: Set this to 2.0.
OAuthAccessTokenURL: Set this to the URL where the JWT is exchanged for an access token.
OAuthJWTCert: Set this to the certificate you want to use. In most cases this will be a path to a PEM or PFX file.
OAuthJWTCertType: Set this to the correct certificate type. In most cases this will either PEMKEY_FILE or PFXFILE.
OAuthJWTCertPassword: If the certificate is encrypted, set this to the encryption password.
OAuthJWTIssuer: Set this to the issuer. This corresponds to the iss field in the JWT.
InitiateOAuth: Set this to GETANDREFRESH.

Note that the JWT signature algorithm cannot be set directly. The connector only supports the RS256 algorithm.

The connector will then construct a JWT including the following fields, and submit it to OAuthAccessTokenURL for an access token.

scope This will come from Scope if it is provided.
aud This will come from OAuthJWTAudience if it is provided.
iss This will come from OAuthJWTIssuer.
iat This is the time when the JWT is generated.
exp This is the value of iat plus the value of OAuthJWTValidityTime.
sub This will come from OAuthJWTSubject if it is provided.

OAuthPassword

AuthScheme: Set this to OAuthPassword.