Destination S3 #
The extracted replicant-cli
will be referred to as the $REPLICANT_HOME
directory in the proceeding steps.
I. Setup Connection Configuration #
-
From
$REPLICANT_HOME
navigate to the sample S3 connection configuration filevi conf/conn/s3.yaml
-
If you store your connection credentials in AWS Secrets Manager, you can tell Replicant to retrieve them. For more information, see Retrieve credentials from AWS Secrets Manager.
Otherwise, you can put your credentials like usernames and passwords in plain form like the sample below:
type: S3 access-key: "REPLICANT" #Replace REPLICANT with the access key of AWS user created from IAM management secret-key: "Replicant#123" #Replace Replicant#123 with the secret Key of the AWS User (note: make sure the specified user has AmazonS3FullAccess) bucket: "bucket_name" #Replace bucket_name with the name of your S3 bucket root: "root_dir" #Replace root_dir with the name of the directory inside the s3 bucket where the CSV files will be stored stage: type: SHARED_FS root-dir: /home/user/stage #Enter the path of the directory where CSV files will be staged before uploading to S3 #file-format: CSV #Possible file formats are CSV or JSON max-connections: 50 #Maximum number of connections Replicant can open in the target s3
II. Set up Applier Configuration #
-
From
$REPLICANT_HOME
navigate to the sample S3 applier configuration filevi conf/dst/s3.yaml
-
If necessary, make the necessary changes as follows:
snapshot: #threads: 16 #max-file-size: 33_554_432 #32MB #delimiter: `,` #CSV files created will have provided delimiter #quote: `”` #escape: `\` #include-header: false #enable or disable toggle column names as header in CSV file #realtime: #threads: 16
III. S3 File Format #
When data is being loaded into S3, the data is first converted to either a CSV or JSON file. To better understand the data format for the CSV and JSON converted files, see S3 File Format.
Important: We highly recommended that you read the S3 File Format page when using S3 as the Target system.