Tasks control the data flow from a source into a destination table. In a Standard Replicate Job, all source tables and views are available to be added as Replicate Tasks to the Job. Select the Add Tables button to view a full table listing in your source. In the new modal, select the specific tables or select all and confirm your selection by clicking Add Selected Tables.
CData Sync will automatically build the Replicate Tasks and save your Job.
If the Sync All option was selected, all available tables and views are automatically added as Replicate Tasks to the Job. Each time the Job is executed, Sync will first search for any new tables or views found in your source and append additional Replicate Tasks to your Job to be included in the run. Use the Exclude button in the Job Settings to remove certain Tasks from the Job and future Job executions will ignore those tables.
Each Task in your Job can be configured individually. Selecting a row will open the Task Settings allowing you to make adjustments to the Task.
Source Table Information
The Source Table Information section gives an overview of the Source Table’s capabilities and key features:
- Primary Key: The column(s) that make up the Primary Key Index for this table.
- Incremental Check Column: The column used by Sync to identify new or modified records during subsequent replication intervals. See Incremental Check Column for more information.
- Captures Deletes: Whether Sync can detect when data has been deleted in the source. See Captures Deletes for more information.
Destination Table Information
The Destination Table Information section gives you full control over where to land your source data.
- Schema Name: The schema name is inherited from the Destination Schema Advanced Job Option. For Destinations that support multiple schemas, select from a list of available schemas in your database.
- Table Name: By default, Sync will create a new table with the same name as the source table in the target destination to load the data into. You have the option of renaming that table by selecting Create New Table. Another option is to map your source table into an existing table in the destination. Select Map to Existing Table to select an existing Destination table to map your source table to. Sync will attempt to auto-map the source to destination columns. Navigate to the Column-Mapping to adjust the default mappings.
The Column-Mapping tab gives you more agency over which data to retrieve from your source table and how you’d like that data to be represented in the target table. See the Transformations section for more information about how to remove or adjust column mappings and apply different transformations to each Replicate Task.
Replication Options control the flow from your Source Application to your target Destination. While you can set Replication Options in your Job that will affect the Job as a whole, Replication Options set in the query will only affect this specific query. By default, each Replication Option set in the Job will be inherited by each query and any Replication Option set in the query will override those values.
Write Custom Queries
The Task Settings will automatically update the Replicate Query as you make changes in the modal. You can also check the Write Custom Query checkbox to manually edit the generated query. See Replicate Command for syntax outlines for REPLICATE and CHECKCACHE along with common examples.
Load From Folder
If the Load all files into single table option was selected, you will need to configure a single LoadFolder task that continuously loads files from a directory or bucket into a single destination table. When running the Job, Sync compares the last modified date of each file in source folder to the value stored in the destination table. To help faciliate this, Sync will add these required system columns to your destination table:
|_lastModified||BigInt||Yes if Modified Action = APPEND|
To adjust the settings for the LoadFolder Tasks, select the Configure button to open the Task settings:
- Modified Action: Whether to APPEND or UPSERT data in your destination table. When a previously loaded file is modified, APPEND will append all rows while UPSERT will replace any existing records.
- File Mask: A pattern to match file names within the folder.
- Schema Name: For Destinations that support multiple schemas, select from a list of available schemas in your database.
- Table Name: By default, Sync will create a new table with the same name as the Job Name in the target destination to load the data into. You have the option of renaming that table by selecting Create New Table.
In the Column Mapping section, you’ll choose to sample a file to build the column mapping from. See the Column Mapping section for more information about how to edit the mappings and apply different transformations to the Task.