Version 23.4.8843

• Add the PostgreSQL Connector
• Authenticate to PostgreSQL
  • Password
  • AzureAD
  • AzurePassword
  • AzureMSI
  • AwsIAMRoles
  • AwsEC2Roles
  • GCPServiceAccount
• Complete Your Connection
• Configure Logical Replication
  • Enable Logical Replication for CData Sync
  • Deletion Behavior
• Support for Incremental Replication
• More Information

You can use the PostgreSQL connector from the CData Sync application to capture data from PostgreSQL and move it to any supported destination. To do so, you need to add the connector, authenticate to the connector, and complete your connection.

Add the PostgreSQL Connector

To enable Sync to use data from PostgreSQL, you first must add the connector, as follows:

1. Open the Connections page of the Sync dashboard.

2. Click Add Connection to open the Select Connectors page.

3. Click the Sources tab and locate the PostgreSQL row.

4. Click the Configure Connection icon at the end of that row to open the New Connection page. If the Configure Connection icon is not available, click the Download Connector icon to install the PostgreSQL connector. For more information about installing new connectors, see Connections.

Authenticate to PostgreSQL

After you add the connector, you need to set the required properties.

• Connection Name - Enter a connection name of your choice.

• Server - Enter the host name or IP of the server that hosts the PostgreSQL database. The default server value is localhost.

CData Sync supports authenticating to PostgreSQL in several ways. Select your authentication method below to proceed to the relevant section that contains the authentication details.

• Password (default)
• AzureAD
• AzurePassword
• AzureMSI
• AwsIAMRoles
• AwsEC2Roles
• GCPServiceAccount

Password

To connect with user credentials, specify the following properties:

• Auth Scheme - Select Password.

• User - Enter the username that you use to authenticate to your PostgreSQL account.

• Password - Enter the password that you use to authenticate to your PostgreSQL account.

AzureAD

To connect with Azure Active Directory, specify the following properties:

• Auth Scheme - Select AzureAD.

• User - Enter the username that you use to authenticate to your PostgreSQL account.

• Azure Tenant - Enter the Microsoft Online tenant to which you want to connect. If you do not specify a tenant, CData Sync uses the default tenant.

• OAuth Client Id - Enter the client Id that you were assigned when you registered your application with an OAuth authorization server.

• OAuth Client Secret - Enter the client secret that you were assigned when you registered your application with an OAuth authorization server.

AzurePassword

To connect with Azure user credentials, specify the following properties:

• Auth Scheme - Select AzurePassword.

• User - Enter the username that you use to authenticate to your PostgreSQL account.

• Password - Enter the password that you use to connect to your PostgreSQL account.

• Azure Tenant - Enter the Microsoft Online tenant to which you want to connect. If you do not specify a tenant, CData Sync uses the default tenant.

AzureMSI

To connect with Azure Managed Service Identity, specify the following properties:

• Auth Scheme - Select AzureMSI.

• User - Enter the username that you use to authenticate to your PostgreSQL account.

AwsIAMRoles

To connect with IAM user credentials, specify the following properties:

• Auth Scheme - Select AwsIAMRoles.

• User - Enter the username that you use to authenticate to your PostgreSQL account.

• AWS Access Key - Enter your Amazon Web Services (AWS) account access key. This value is available on you AWS security-credentials page.

• AWS Secret - Enter your Amazon Web Services (AWS) account secret. This value is available on you AWS security-credentials page.

• AWS Role Arn - Enter the Amazon Resource Name of the role that you want to use when you authenticate.

• AWS External Id (optional) - Enter a unique identifier that might be required when you assume a role in another account.

AwsEC2Roles

This auth scheme does not require any additional settings. So, you can select this scheme and continue with the settings and steps in Complete Your Connection.

GCPServiceAccount

To connect with a GCP service account, specify the following properties:

• Auth Scheme - Select GCPServiceAccount.

• OAuth JWT Cert - Enter your Java web tokens (JWT) certificate store.

• OAuth JWT Cert Type - Enter the type of key store that contains your JWT Certificate. The default type is PEMKEY_BLOB.

• OAuth JWT Cert Password (optional) - Enter the password for your OAuth JWT certificate.

• OAuth JWT Cert Subject (optional) - Enter the user subject for which the application is requesting delegated access.

Complete Your Connection

To complete your connection:

1. Specify these settings:

   • Database - Enter the name of the database that to which you want to connect when you connect to the PostgreSQL server.

   • Port - Enter the port number of the server that hosts the PostgreSQL database. The default port value is 5432.

2. Define advanced connection settings on the Advanced tab. (In most cases, though, you should not need these settings.)

3. If you authenticate with AzureAD, AzurePassword, AzureMSI, or GCPServiceAccount, click Connect to PostgreSQL to connect to your PostgreSQL account.

4. Click Create & Test to create your connection.

Configure Logical Replication

CData Sync can use logical replication that is configured in PostgreSQL to update your destination table incrementally. PostgreSQL uses logical decoding to surface the contents of the write-ahead logs, which track data changes in the database, into a readable format. Those changes are read by Sync and pushed into the destination.

Enable Logical Replication for CData Sync

To enable logical replications for Sync:

1. Ensure that the wal_level=logical parameter is set in your PostgreSQL database. You can set this parameter in the postgresql.conf file.

2. Create a logical replication slot for Sync by submitting the following SELECT statement:

   SELECT pg_create_logical_replication_slot('cdatasync_replication_slot', 'test_decoding');
   

   Note: The slot name cdatasync_replication_slot is an example slot name. You can substitute any name in its place. You need this slot name when you create your job in Sync. It is important that this slot is used by Sync only because changes are consumed when it reads from the slot.

3. Ensure that the user that connects from Sync has permission to read the replication slot by submitting the following statement.

   ALTER ROLE <em>PostreSQL-User</em> WITH REPLICATION;
   

4. Create a job in Sync with a PostgreSQL source. After you select the Change Data Capture option, enter the replication slot name.

Deletion Behavior

Logical replication enables Sync to track deleted records when the source table contains a primary key. If the source table does not contain a primary key, Sync cannot retrieve deleted records.

Support for Incremental Replication

The PostgreSQL source supports incremental replication. Incremental replication reduces the workload tremendously and minimizes bandwidth use and latency of synchronization. Moving data in increments offers great flexibility when you are dealing with slow APIs or daily quotas.

You must configure the incremental check column based on the PostgreSQL source columns. An incremental check column is either a datetime or integer-based column that Sync uses to identify new or modified records when it replicates data. Therefore, you must configure incremental check on a datetime or integer-based column in your PostgreSQL table for incremental replication to occur.

For details about how to set up incremental replication, see Incremental Replication.

More Information

For more information about interactions between CData Sync and PostgreSQL, see PostgreSQL Connector for CData Sync.