Destination SingleStore #
The extracted replicant-cli
will be referred to as the $REPLICANT_HOME
directory in the proceeding steps.
Prerequisites #
You must have a user configured in SingleStore for replication with SELECT
, INSERT
, UPDATE
, DELETE
, CREATE
, DROP
permissions on application databases.
If SingleStore user does not have create database permission then you must create a database named io_replicate
and provide SELECT
, INSERT
, UPDATE
, DELETE
, CREATE
, DROP
privileges to SingleStore user.
I. Setup Connection Configuration #
-
From
$REPLICANT_HOME
, navigate to the sample SingleStore connection configuration file:vi conf/conn/singlestore.yaml
-
Make the necessary changes as follows:
type: SINGLESTORE host: localhost #Replace localhost with address to your SingleStore host port: 3306 #Replace default port 3306 if needed username: 'replicant' #Replace replicant with your SingleStore user password: 'Replicant#123' #Replace Replicant#123 with your user's password max-connections: 30 #Maximum number of connections replicant can open in SingleStore max-retries: 10 retry-wait-duration-ms: 1000
II. Setup Applier Configuration #
Edit the applier configurations if required.
-
From
$REPLICANT_HOME
, navigate to the sample Applier Configuration File:vi conf/dst/singlestore.yaml
-
Make the necessary changes as follows:
snapshot: table-store: COLUMN #Applied to all tables; can be overridden for certain tables if needed in the per-table-config section below per-table-config: catalog: tpch tables: singlestore_orders: #Replace this with the name of the specific table you are configuring for in SingleStore table-store: COLUMN #Enter the table's store (ROW/COLUMN etc.) sort-key: [singlestore_orderkey] #If applicable, replace singlestore_orderkey with a list of columns to be created as the sort key shard-key: [c2] #If applicable, replace c2 with a list of columns to be created as the shared key realtime: #replay-shard-key-update-as-delete-insert: true #retry-failed-txn-idempotently: true
-
Below is a sample applier file with commonly used configuration parameters:
snapshot: threads: 32 # batch-size-rows: 10_000 # txn-size-rows: 1_000_000 bulk-load: enable: true type: FILE #native-load-configs: "ERRORS HANDLE 'singlestore_load_error'" #User provided SingleStore LOAD configs. These will be appended to the generated LOAD SQL command. #table-store: ROW #init-constraint-post-snapshot: false per-table-config: - catalog: tpch tables: singlestore_orders: table-store: COLUMN sort-key: [singlestore_orderkey] #shard-key: [c2] partsupp: #table-type: REGULAR | REFERENCE table-store: ROW partsupp_macro_delta: table-store: COLUMN sort-key: [partkey, suppkey] realtime: threads: 8 txn-size-rows: 10000 batch-size-rows: 1000 _oper-queue-size-rows: 20000 # replay-shard-key-update-as-delete-insert: true # retry-failed-txn-idempotently: true
For a detailed explanation of configuration parameters in the applier file, read: Applier Reference