Destination MinIO #
From version 23.04.30.4, Arcion supports MinIO as a S3-compatible data target. Follow the instructions in this page to set up MinIO as the replication target in Arcion Replicant.
The following steps refer the extracted Arcion self-hosted CLI download as the
S3 file format #
When Replicant loads data into S3, Replicant first converts the data to either a CSV or a JSON file. To better understand the data format for the CSV and JSON converted files, see Arcion Internal CDC Format for Amazon S3. We highly recommended that you read the Arcion Internal CDC Format for Amazon S3 page when using S3 as the target system.
I. Set up connection configuration #
Specify your MinIO connection details to Replicant with a connection configuration file. You can find a sample connection configuration file
minio.yaml in the
type: S3 endpoint: service-endpoint: SERVICE_ENDPOINT #service endpoint either with or without the protocol (e.g. https://sns.us-west-1.amazonaws.com or sns.us-west-1.amazonaws.com) signingRegion: "SIGNING_REGION" #region to use for SigV4 signing of requests (e.g. us-west-1) access-key: "YOUR_MINIO_ACCESS_KEY" secret-key: "YOUR_MINIO_SECRET_KEY" bucket: "MINIO_BUCKET_NAME" root: "ROOT_PATH_UNDER_BUCKET" stage: type: SHARED_FS root-dir: PATH_TO_STAGE_DIRECTORY #directory where CSV files will be staged before uploading to S3 max-connections: 50
Replace the following:
SERVICE_ENDPOINT: The service endpoint with or without the protocol—for example,
SIGNING_REGION: The region to use for AWS Signature Version 4 requests—for example,
YOUR_MINIO_ACCESS_KEY: The MinIO access key for the MinIO user.
YOUR_MINIO_SECRET_KEY: The MinIO secret key for the MinIO user.
MINIO_BUCKET_NAME: The MinIO bucket name.
ROOT_PATH_UNDER_BUCKET: The root path under MinIO bucket. Replicant creates all data files under
MINIO_BUCKET_NAME/ROOT_PATH_UNDER_BUCKET. For example, if you set
replicant/s3dst, Replicant creates the data files under
PATH_TO_STAGE_DIRECTORY: Directory where Replicant stages CSV files before uploading them to S3.
II. Set up Applier configuration #
Follow the instructions in Applier configuration for Amazon S3.