Use Type-2 CDC #

From version 22.07.19.3 onwards, Arcion supports Type-2 CDC for Snowflake as the Target.

Overview #

Type-2 CDC enables a Target to have a history of all transactions performed in the Source. For example:

• An INSERT in the Source is an INSERT in the Target.
• An UPDATE in the Source is an INSERT in the Target with additional metadata like Operation Performed, Time of Operation, etc.
• A DELETE in the Source is an INSERT in the Target: INSERT with OPER_TYPE as DELETE.

Arcion supports the following metadata related to source-specific fields:

• query_timestamp: Time at which the user on Source fired a query.
• extraction_timestamp: Time at which Replicant detected the DML from logs.
• OPER_TYPE: Type of the operation (INSERT/UPDATE/DELETE).

The primary requirement for Type-2 CDC is to enable full row logging in the Source.

Note: Support for Type-2 CDC is limited to the following cases:

• Sources that support CDC.
• realtime and full modes.

To enable Type-2 CDC for your Snowflake target, follow the steps below:

1. Add the following two parameters under the realtime section of the Snowflake Applier configuration file:

   realtime:
       enable-type2-cdc: true
       replay-strategy: NONE
   

2. In the Extractor configuration file of Source, add the following parameter under the snapshot section:

   snapshot:
       csv-publish-method: READ
   

For a detailed explanation of configuration parameters in the Applier file, see Applier Reference.