Mapper Configuration #
While replicating data between storages of different type, Replicant by default attempts to transfer the data as fetched from the source while maintaining its structure. But there maybe situations where you need more control over Source data mapping. That’s where a Mapper file comes in.
By using the Mapper file, you can precisely define how the data retrieved from the Source is applied to the Target/Destination.
You can specify the Mapper file when running Replicant in the folliwng way :
./bin/replicant snapshot conf/conn/oracle.yaml \ conf/conn/singlestore.yaml \ --map sqlserver_to_singlestore_map.yaml
The Mapper file contains a map of rules where each rule applies to a single Target catalog or schema (namespace).
- For databases that support both catalog and schema, each rule applies to a single schema. The schema must be prefixed with the catalog (fully qualified).
- For each Target namespace, it’s possible to define a list of Source namespaces, the contents of which will then be mapped into the Target namespace.
When you need more control, you can define additional rules for the tables and columns. Table rules are defined in the similar fashion as namespace rules—using the tables map. In the tables map, you can specify multiple source tables for each destination tables.
Note: Target tables are defined using only their name, while Source table names need to be fully qualified with each table’s respective catalog and schema.
For each table mapping, it’s also possible to map columns based on their names.
Structure of Mapper Rules #
Following is a template of the structure of Mapper rules:
rules: [<target_namespace>]: source: - [<source_namespace>]: . . . tables: <target_table_name>: source: - [<src_namespace>, <src_table_name>]: <target_col_name>: <src_col_name> . . . . . . . . . . . .
Below is a sample Mapper file for a SQL Server-to-SingleStore pipeline:
rules: [testdb_s]: source: - [testdb, dbo]: tables: testTB1_s: source: [testdb, dbo, TestTB1]: col1_s: col1
In the above example:
testdb_sis the Target database name that you want to change to.
sourceparameter, the flow sequence containing
dborepresents Source database name and schema name (SQL Server default schema name is
testTB1_sis the name of the table at Target that you want to change to.
- The flow sequence
[testdb, dbo, TestTB1]under
sourcerepresents the Source database name, Source schema name, and Source table name respectively. Pay attention to case sensitivity.
- The key-value pair
col1_s: col1maps Target column name to Source column name
You can also find sample Mapper files for different pipelines in the
mapper directory of your Arcion Self-hosted download.
Dynamic trimming options #
There maybe cases where the Target database does not support the name length of tables in the Source database. In those cases, you can make use of the following options inside a mapper file to enable trimming of table and column names:
When set to
true, table or column names are trimmed so that it fits the length the target supports.
Represents the namespace where the mappings of table or column are stored in the target database.
catalog: Specify catalog name if applicable.
schema: Specify schema name if applicable.
Convert letter case for database objects #
In cases where the target database requires database object names (tables or columns) to be in particular letter case, you can use the
convert-case option inside the mapper file:
The letter case to use for table or column names while mapping. The following values are allowed:
When set to
UPPERCASE, Replicant converts table or column names to the appropriate letter case while mapping. When set to
DEFAULT, Replicant maintains the letter case of source object names.
Note: If the table is explicitly mapped, then that mapping will override the convert-case behaviour. If a source table named
REGIONis mapped to destination table
region_lowercase, then this mapping will override the
convert-caserule for this table. So, on target the table name becomes
region_lowercaseinstead of uppercase