Destination S3 #
replicant-cli will be referred to as the
$REPLICANT_HOME directory in the proceeding steps.
I. Setup Connection Configuration #
$REPLICANT_HOMEnavigate to the sample S3 connection configuration file
Make the necessary changes as follows:
type: S3 access-key: "REPLICANT" #Replace REPLICANT with the access key of AWS user created from IAM management secret-key: "Replicant#123" #Replace Replicant#123 with the secret Key of the AWS User (note: make sure the specified user has AmazonS3FullAccess) bucket: "bucket_name" #Replace bucket_name with the name of your S3 bucket root: "root_dir" #Replace root_dir with the name of the directory inside the s3 bucket where the CSV files will be stored stage: type: SHARED_FS root-dir: /home/user/stage #Enter the path of the directory where CSV files will be staged before uploading to S3 #file-format: CSV #Possible file formats are CSV or JSON max-connections: 50 #Maximum number of connections Replicant can open in the target s3
Setup Applier Configuration #
$REPLICANT_HOMEnavigate to the sample S3 applier configuration file
If necessary, make the necessary changes as follows:
snapshot: #threads: 16 #max-file-size: 33_554_432 #32MB #delimiter: `,` #CSV files created will have provided delimiter #quote: `”` #escape: `\` #include-header: false #enable or disable toggle column names as header in CSV file #realtime: #threads: 16
III. S3 File Format #
When data is being loaded into S3, the data is first converted to either a CSV or JSON file. Follow the link below to understand the data format for the CSV and JSON converted files.
Note: It is highly recommended that you read the S3 file format page when using S3 as the target system.