IBM Informix

Source IBM Informix #

The extracted replicant-cli will be referred to as the $REPLICANT_HOME directory in the proceeding steps.

I. Set up CDC Replication #

For enabling CDC-based replication from the source Informix server, please follow the instructions in Enabling CDC Replication for Informix.

II. Set up Logical Log Configuration #

For setting up logical log configuration, follow the instructions in Logical Log Configuration Guidelines.

III. Create the Heartbeat Table #

For CDC replication, you must create the heartbeat table on the source database with the following DDL:

Remember to grant INSERT, UPDATE, and DELETE privileges for this table to the user that you provided to Replicant.

CREATE TABLE tpch:tpch.replicate_io_cdc_heartbeat(timestamp INT8 NOT NULL, PRIMARY KEY(timestamp) CONSTRAINT cdc_heartbeat_id_repl1_repl1) 
LOCK MODE ROW

IV. Set up Connection Configuration #

  1. From $REPLICANT_HOME, navigate to the sample connection configuration file:

    vi conf/conn/informix.yaml

  2. If you store your connection credentials in AWS Secrets Manager, you can tell Replicant to retrieve them. For more information, see Retrieve credentials from AWS Secrets Manager.

    Otherwise, you can put your credentials like usernames and passwords in plain form like the sample below:

    type: INFORMIX

host: localhost #Replace localhost with your Informix server's hostname
port: 9088  # In case of SSL connection use SSL port

server: 'informix'
database: 'tpch' # Name of the catalog from which the tables are to be replicated

username: 'informix' #Replace informix with the user that connects to your Informix server
password: 'in4mix' #Replace in4mix with your user's password 
informix-user-password: 'in4mix' #Password for the "informix" user, required for performing CDC. Not required in snapshot replication.

max-connections: 15
max-retries: #Number of times any operation on the source system will be re-attempted on failures.

#ssl:
#  trust-store: 
#    enable: true
#    path: "/home/informix/ssl/truststore.jks" #Path to the JKS truststore containing the trust certificate of the Informix server
#    password: "in4mix" #The truststore password

    • You have to connect to the syscdcv1 catalogue on the server as the user informix to be able to use Change Data Capture (CDC). The informix-user-password parameter of the config file above should have the password of user informix. For more information, see Preparing to use the Change Data Capture API on IBM Informix Documentation.

    • You can use SSL to connect to the Informix server using the configuration parameters shown above. To konw about Informix server side SSL setup, see Configuring Informix server instance for SSL connections on IBM Informix Documentation.

V. Set up Extractor Configuration #

  1. From $REPLICANT_HOME, navigate to the Extractor configuration file:

    vi conf/src/informix.yaml

  2. The configuration file has two parts:

    • Parameters related to snapshot mode.
    • Parameters related to realtime mode.

    For snapshot mode, make the necessary changes as follows:

    snapshot:
  threads: 16
  fetch-size-rows: 5_000

#  lock:
#    enable: true
#    scope: TABLE   # DATABASE, TABLE
#    force: false
#    timeout-sec: 5

#  min-job-size-rows: 1_000_000
#  max-jobs-per-chunk: 32

  split-method: MODULO  # Allowed values are RANGE, MODULO
  per-table-config:
  - catalog: "tpch"
    schema: "tpch"
    tables:
      lineitem:
        split-key: "l_orderkey"
#       row-identifier-key: ["l_linenumber", "l_orderkey"]
#       split-method : MODULO  # Table specific overridable config : allowed values are RANGE, MODULO
#       num-jobs: 1
      orders:
        split-key: "o_orderkey"
#       num-jobs: 1

    For operating in realtime mode, use the realtime section to specify your configuration. The following parameters are specific to Informix:

    • db-charset [v22.02.12.23]: JDK equivalent of Informix code set. For example, UTF-8, ISO-8859-1 etc.
    • start-position: This parameter allows using user specified sequence number or timestamp to start reading Change Data Capture operations.

      • sequence-number: Specifies the start position for the Change Data Capture.

      • timestamp-ms: Optional parameter. Used as a filter—any transaction with commit timestamp older than this is ignored. Should be used together with sequence-number for finer tuning.

      • create-checkpoint: true or false. If set to true, it does two things:

        a. Obtains the current sequence-number from the Informix server and logs it in the trace.log file. The log has the following format:

        Successfully created replication checkpoint at CDC sequence number: <seq_number>

        b. Sets up a checkpoint, from which you can resume the replication at any point. You can do so by appending the --resume argument to the replication command that created the checkpoint.

    The following is a sample configuration for realtime mode:

    realtime:
  threads: 4
  fetch-size-rows: 256
  _buffer-size: 1000
  _read-timeout: 3

  db-charset: UTF-8

  start-position:
  # create-checkpoint: false
  # sequence-number: 51553788137
  # timestamp-ms: 1647378865000

  heartbeat:
    enable: true
    catalog: tpch
    schema: tpch
    interval-ms: 10000

    With Informix as Source, you have the option to use the Source Column Transformation feature of Replicant for the following pipelines:

    • Informix to PostgreSQL
    • Informix to Kafka

    For more information, see Source Column Transformation.

    Example of creating a checkpoint and resuming replication #

    Let’s assume you used the following command to start the replication, with your realtime configuration same as above:

    ./bin/replicant real-time conf/conn/informix.yaml conf/conn/postgresql.yaml \
--filter filter/informix_filter.yaml \
--extractor conf/src/informix.yaml \
--replace

    In the trace.log file, you wil find the sequence-number corresponding to this replication job.

    With a valid sequence-number and create-checkpoint set to true, you can resume replication from the position of the most recently created checkpoint (which is the checkpoint you created via the previous command) in the following way:

    ./bin/replicant real-time conf/conn/informix.yaml conf/conn/postgresql.yaml \
--filter filter/informix_filter.yaml \
--extractor conf/src/informix.yaml \
--replace --resume

For a detailed explanation of configuration parameters in the Extractor file, read Extractor Reference.