Enabling SSIS in Visual Studio 2022

If you're using Visual Studio 2022, you will need to install the SQL Server Integration Services Projects extension to use SSIS.

1. Navigate to Extensions > Manage Extensions.
2. In the Manage Extensions window's search box, search for "SQL Server Integration Services Projects 2022" and select the extension in the list.
3. Click Download.
4. Close Visual Studio and run the downloaded Microsoft.DataTools.IntegrationServices.exe installer. Proceed through the installer with default settings.
5. Open Visual Studio. There should now be an "Integration Services Project" project template available.

Adding the NetSuite Connection Manager

Create a new connection manager as follows:

1. Create a Visual Studio project with the "Integration Services Project" template.
2. In the project, right-click within the Connection Managers window and select New Connection from the menu.
3. In the Description column, select CData NetSuite Connection Manager and click Add...
4. Configure the component as described in the next section.

Alternatively, if you have an existing project and CData NetSuite Source or CData NetSuite Destination:

1. Right-click your CData NetSuite source or destination component in your data flow
2. Select Edit... to open an editor window.
3. Click the New... button next to the Connection manager: dropdown selector to create a connection manager.
4. Configure the component as described in the next section.

Connecting to NetSuite

NetSuite currently offers two distinct APIs:

• SuiteTalk is the older SOAP-based service we use to communicate with NetSuite. It has broad support for a lot of entities and full support for INSERT/UPDATE/DELETE. However, the tools offered for selecting data are fairly weak, which yields very poor performance during a SELECT. There is not a great way to join tables. Grouping and aggregating data is unavailable with this API, which means to support them they must be done entirely client side.
• SuiteQL is the newer API. It offers a SQL-like method of communicating with the service, which allows for more rich JOIN support, as well as GROUP BY and aggregations. It also fully supports retrieving only the columns you want to select. This makes it much more performant than SuiteTalk for selecting data. However, it only supports selecting data.

To connect to NetSuite, you must:

1. Set Schema to specify the API you are using to connect.
   If you are just retrieving data, we recommend you use SuiteQL. If you need to both retrieve and modify data, we recommend you use SuiteTalk.
2. Set the appropriate connection options for the API in use. (Be aware that each API has different available connection options, as described in Permission Configurations.)
3. If you plan to connect to SuiteTalk, we recommend that you specify the NetsuiteMetadataFolder. The Netsuite Metadata Folder is a folder where NetSuite metadata files are stored. If you do not specify one, a folder location is selected automatically. To ensure fast load times when listing metadata about tables, it is best to set this property.
4. Ensure that any connecting user has permissions on the specified AccountId to connect through NetSuite Web services. For details about required and optional permissions, see Permission Configurations.
   
   

   To create or edit a role for Web services permissions:

   1. Log into NetSuite.
   2. If you need to create a new role, navigate to Setup > User/Roles > Manage Roles > New.
      If you need to modify an existing role, navigate to Setup > User/Roles > Manage Roles and select the role to be edited.
   3. Click Permissions > Setup.
   4. Add permissions for SOAP Web Services and REST Web Services.
   5. Add other permissions that are needed for interacting with various entities and transactions. (See Permission Configurations.)
   6. Under Setup, navigate to User/Roles > Manage Users. Select the user you just created or edited.
   7. At the Access tab, add the newly-created role.
   8. Save the new or changed user.

Authenticating to NetSuite

This section describes how to authenticate to NetSuite from earlier versions of SuiteTalk (which use Basic authentication), later versions of SuiteTalk, and all versions of Suite QL.

Suite Talk 2020.2 or Earlier

As of 2020.2, NetSuite no longer supports Basic (user/password) authentication. While the CData SSIS Components for NetSuite continues to support User/Password connections for Version set to values lower than 2020.2, we recommend that all customers migrate to the OAuth and Token Based Authentication mechanisms described below.

To continue using Basic authentication with an early version of SuiteTalk, set the AuthScheme to Basic to support user/password credentials. However, you must also manually specify a lower Version to use it.

To support either Token-Based or OAuth 2.0 authentication, set the AuthScheme to OAuth.

Suite Talk or Suite QL

NetSuite offers two forms of OAuth authentication:

• Token Based Authentication (TBA) is essentially OAuth 1.0 with the OAuthAccessToken and OAuthAccessTokenSecret created within the NetSuite UI instead of at runtime. TBA is available for both SuiteTalk versions later than 2020.2, and SuiteQL.
• OAuth 2.0 Authentication is only available for SuiteQL. To enforce OAuth 2.0 authentication, you can do one of the following:
  • Set OAuthVersion explicitly to the API in use, or
  • Set Schema to SuiteQL.

Note: Be aware that if you are using SuiteTalk, you will not be able to use OAuth 2.0 even if you configure the OAuthVersion to SuiteTalk. SuiteTalk does not support OAuth 2.0.

To support either Token-Based or OAuth 2.0 authentication, set the AuthScheme to OAuth.

Token Based Authentication (OAuth 1.0)

Token Based Authentication (TBA) is essentially OAuth 1.0. You can use TBA with either SuiteTalk or the SuiteQL Schema.

Token Based Authentication is performed by creating the OAuthClientId, OAuthClientSecret, OAuthAccessToken and OAuthAccessTokenSecret directly within the NetSuite UI by an administrator with permissions to do so.

To support the use of TBA, set the AuthScheme to Token and create tokens in the NetSuite UI, as described below.

Creating Tokens in the NetSuite UI

Many NetSuite admins prefer to create and assign a token directly in the NetSuite UI. Connecting via token enables users to bypass the normal steps for generating an OAuth Access Token. This provides the admin with more direct control over giving access, although it always requires manual steps to be taken in the UI each time they need to create a new token.

Instead of creating and assigning a token directly in the NetSuite UI, follow this procedure:

1. In NetSuite, log in using an account with administrator privileges.
2. Navigate to Setup > Company > Enable Features > SuiteCloud > Manage Authentication.
3. Select Token-Based Authentication and TBA: Authorization Flow.
4. Save changes.
5. Navigate to Setup > Integration > Manage Integrations.
6. Create a new integration and select Token-Based Authentication.
   When the integration is created, it displays the Consumer Key and Consumer Secret. These map directly to the OAuthClientId and OAuthClientSecret connection properties.
7. Record the values for Consumer Key and Consumer Secret.
8. To create a new token role or modify an existing role, navigate to Setup > User/Roles > Manage Roles.
9. Navigate to Permissions > Setup.
10. Assign the role User Access Token: Full, Access Token Management: Full, and Web Services: Full permissions.
11. To add the new token role to a user who is listed in Lists > Employees > Employees, select the employee, and then navigate to Access > Roles.
12. Navigate to Setup > User/Roles > Access Tokens and create a new access token.
13. For application name, specify the integration that was created earlier. Assign this to the same user and role that was created or updated in the previous steps.
14. After creating the access token, a Token Id and Token Secret are displayed. These map directly to the OAuthAccessToken and OAuthAccessTokenSecret. Write them down.

After the access token is created, it can be used to connect using the values obtained from the previous steps. Before you connect, specify at least the following connection properties:

• AccountId = the account to connect to.
• OAuthClientId = the Consumer Key displayed when the application was created.
• OAuthClientSecret = the Consumer Secret displayed when the application was created.
• OAuthAccessToken = the Token Id when the access token was created.
• OAuthAccessTokenSecret = the Token Secret when the access token was created.

OAuth 2.0

NetSuite SuiteTalk supports OAuth 2.0 authentication. To enable this authentication from all OAuth flows, you must set AuthScheme to OAuth. If you want to authenticate via a Web application, you must create a custom OAuth application, as described in Creating a Custom OAuth Application.

The following subsections describe how to authenticate to NetSuite from three common authentication flows:

• Desktop: a connection to a server on the user's local machine, frequently used for testing and prototyping. Authenticated via either embedded OAuth or custom OAuth.
• Web: access to data via a shared website. Authenticated via custom OAuth only.
• Headless Server: a dedicated computer that provides services to other computers and their users, which is configured to operate without a monitor and keyboard. Authenticated via embedded OAuth or custom OAuth.

For information about how to create a custom OAuth application, and why you might want to create one even for auth flows that have embedded OAuth credentials, see Creating a Custom OAuth Application.

For a complete list of connection string properties available in NetSuite, see Connection.

Desktop Applications

CData provides an embedded OAuth application that simplifies authentication at the desktop; that is, in situations where the user is using a local server not connected to the internet.

You can also authenticate from the desktop via a custom OAuth application, which you configure and register at the NetSuite console. For further information, see Creating a Custom OAuth Application.

Before you connect, set the following variables:

• InitiateOAuth = GETANDREFRESH. Used to automatically get and refresh the OAuthAccessToken.
• Custom OAuth applications only:
  • OAuthClientId = the client Id assigned when you registered your custom OAuth application.
  • OAuthClientSecret = the client secret assigned when you registered your custom OAuth application.
  • CallbackURL = the redirect URI defined when you registered your custom OAuth application.

When you connect, the component opens NetSuite's OAuth endpoint in your default browser. Log in and grant permissions to the application.

After you grant permissions to the application, the component completes the OAuth process:

1. The component obtains an access token from NetSuite and uses it to request data.
2. The OAuth values are saved in the path specified in OAuthSettingsLocation. These values persist across connections.

When the access token expires, the component refreshes it automatically.

Automatic refresh of the OAuth access token:

To have the component automatically refresh the OAuth access token, do the following:

1. The first time you connect to data, set the following connection parameters:
   • InitiateOAuth = REFRESH.
   • OAuthClientId = the client Id in your application settings.
   • OAuthClientSecret = the client secret in your application settings.
   • OAuthAccessToken = the access token returned by GetOAuthAccessToken.
   • OAuthSettingsLocation = the path where you want the component to save the OAuth values, which persist across connections.
2. On subsequent data connections, set the following:
   • InitiateOAuth
   • OAuthSettingsLocation

Manual refresh of the OAuth access token:

The only value needed to manually refresh the OAUth access token is the OAuth refresh token.

1. To manually refresh the OAuthAccessToken after the ExpiresIn period (returned by GetOAuthAccessToken) has elapsed, call the RefreshOAuthAccessToken stored procedure.
2. Set the following connection properties:
   • OAuthClientId = the Client Id in your application settings.
   • OAuthClientSecret = the Client Secret in your application settings.

3. Call RefreshOAuthAccessToken with OAuthRefreshToken set to the OAuth refresh token returned by GetOAuthAccessToken.
4. After the new tokens have been retrieved, set the OAuthAccessToken property to the value returned by RefreshOAuthAccessToken. This opens a new connection.

Store the OAuth refresh token so that you can use it to manually refresh the OAuth access token after it has expired.

Headless Machines

If you need to log in to a resource that resides on a headless machine, you must authenticate on another device that has an internet browser. You can do this in either of the following ways:

• Option 1: Obtain the OAuthVerifier value.
• Option 2: Install the component on a machine with an internet browser and transfer the OAuth authentication values after you authenticate through the usual browser-based flow.

After you execute either Option 1 or Option 2, configure the driver to automatically refresh the access token on the headless machine.

Option 1: Obtaining and Exchanging a Verifier Code

To obtain a verifier code, you must authenticate at the OAuth authorization URL. Do the following:

1. Authenticate from the machine with an internet browser, and obtain the OAuthVerifier connection property.

   If you are using the embedded OAuth application, call the GetOAuthAuthorizationURL stored procedure. Open the URL returned by the stored procedure in a browser.

   If you are using a custom OAuth application, set the following properties:

   • InitiateOAuth = OFF.
   • OAuthClientId = the client Id assigned when you registered your application.
   • OAuthClientSecret = the client secret assigned when you registered your application.

2. Call the GetOAuthAuthorizationURL stored procedure. The stored procedure returns the CallbackURL established when the custom OAuth application was registered. (See Creating a Custom OAuth Application.)

   Copy this URL and paste it into a new browser tab.

3. Log in and grant permissions to the component. The OAuth application redirects you the redirect URI, with a parameter called code appended. Note the value of this parameter; you will need it later, to configure the OAuthVerifier connection property.

4. Exchange the OAuth verifier code for OAuth refresh and access tokens. On the headless machine, set the following connection properties to obtain the OAuth authentication values:

   • InitiateOAuth = REFRESH.
   • OAuthVerifier = the noted verifier code (the value of the code parameter in the redirect URI).
   • OAuthSettingsLocation = persist the encrypted OAuth authentication values to the specified file.
   • Custom OAuth applications only:
     • OAuthClientId = the client Id in your custom OAuth application settings.
     • OAuthClientSecret = the client secret in the custom OAuth application settings.

5. Test the connection to generate the OAuth settings file.

6. After you re-set the following properties, you are ready to connect:

   • InitiateOAuth = REFRESH.
   • OAuthSettingsLocation = the file containing the encrypted OAuth authentication values. To enable the automatic refreshing of the access token, be sure that this file gives read and write permissions to the component.
   • Custom OAuth applications only:
     • OAuthClientId = the client Id assigned when you registered your application.
     • OAuthClientSecret = the client secret assigned when you registered your application.

Option 2: Transferring OAuth Settings

Prior to connecting on a headless machine, you must install and create a connection with the driver on a device that supports an internet browser. Set the connection properties as described above in "Desktop Applications".

After completing the instructions in "Desktop Applications", the resulting authentication values are encrypted and written to the path specified by OAuthSettingsLocation. The default filename is OAuthSettings.txt.

Test the connection to generate the OAuth settings file, then copy the OAuth settings file to your headless machine.

To connect to data via the headless machine, set the following connection properties:

• InitiateOAuth = REFRESH
• OAuthSettingsLocation = the path to the OAuth settings file you copied from the machine with the browser. To enable automatic refreshing of the access token, ensure that this file gives read and write permissions to the component.
• Custom OAuth applications only:
  • OAuthClientId = the client Id assigned when you registered your custom OAuth application.
  • OAuthClientSecret = the client secret assigned when you registered your custom OAuth application.

Concurrent Requests

NetSuite allows only a certain number of concurrent requests per account, which is configurable per integration connection. (The default is usually 5.) If another request is made when the maximum number of concurrent requests is already in use, the user may receive an "Only one request may be made against a session at a time" error.

The CData SSIS Components for NetSuite attempts to account for this situation by staggering additional requests so as not to exceed the concurrent request limit. However, if an account is being connected to from multiple machines or applications, it is not always possible to do this correctly.

Asynchronous Services

Slow NetSuite response times extend to INSERTs, UPDATEs, and DELETEs. This can be especially noticeable if the BatchMode property on the CData SSIS Components for NetSuite Destination is set to false.

When inserting, updating, or deleting multiple records at the same time, it may be worthwhile to set UseAsyncServices to true. This causes the request to be processed asynchronously on NetSuite's end and causes a JobId to be returned in the BulkJobs variable.

Each JobId can be checked against the stored procedures CheckJobStatus and GetJobResults for information about when the job is completed, if any errors occurred, and for the InternalIds for newly created entities.