MongoDB

Destination MongoDB #

The extracted replicant-cli will be referred to as the $REPLICANT_HOME directory.

I. Set up Connection Configuration #

  1. From $REPLICANT_HOME, navigate to the connection configuration file:

    vi conf/conn/mongodb_dst.yaml
    
  2. Make the necessary changes as follows:

    type: MONGODB
    
    url: "mongodb://localhost:27018/?w=majority" #MongoDB connection URL
    
    max-connections: 30 #max number of connections Replicant can open in MongoDB
    
    #ssl:
    #  key-store:
    #    path: '/path/to/key/store'
    #    password: 'ChangePassword'
    #  trust-store:
    #    path: '/path/to/trust/store'
    #    password: 'ChangePassword'
    

II. Set up Applier Configuration #

  1. From $REPLICANT_HOME, navigate to the Applier configuration file:

    vi conf/dst/mongodb.yaml
    
  2. The configuration file has two parts:

    • Parameters related to snapshot mode.
    • Parameters related to realtime mode.

    For snapshot mode, make the necessary changes as follows:

    snapshot:
      threads: 16
    
      batch-size-rows: 5000
      txn-size-rows: 5000
    #  map-key-to-id: false
    #  skip-tables-on-failures : false
      bulk-load:
        enable: false
        type: FILE   # PIPE, FILE
      handle-failed-opers: true
      initIndexesPostSnapshot: true
    #   denormalize:
    #     enable: true
    #  user-role:
    #    init-user-roles: true
    #  init-system-tables: true
    

    For operating in realtime mode, use the realtime section to specify your configuration. Below is a sample config:

    realtime:
      threads: 8
      batch-size-rows: 1000
      txn-size-rows: 2_0000
      handle-failed-opers: true
    #  map-key-to-id: false
    #  skip-tables-on-failures : false
    #   perTableConfig:
    #   - schema: tpch
    #     tables:
    #       CUSTOMER:
    #         skip-upto-cursor: '{"extractorId":0,"replicaSetName":"mongors1","resumeToken":6868517489379115009,"seqNum":3,"v":1,"timestamp":1599201348000}'
    
    # Transactional mode config
    # realtime:
    #   threads: 1
    #   batch-size-rows: 1000
    #   replay-consistency: GLOBAL #allowed values are GLOBAL/EVENTUAL
    #   txn-group-count: 100
    #   skip-upto-cursors: ['{"extractorId":0,"replicaSetName":"mongors1","resumeToken":6868517489379115009,"seqNum":3,"v":1,"timestamp":1599201348000}']
    
    

For a detailed explanation of configuration parameters in the Applier file, read Applier Reference.