Destination local storage #

Arcion supports writing data locally to your disk in CSV format instead of sending it to an external target database. This page contains instructions on how to use your local storage as a real time destination for different data sources.

The following steps refer to the extracted Arcion self-hosted CLI download as the $REPLICANT_HOME directory.

I. Set up connection configuration #

Specify the connection details to Replicant with a connection configuration file. You can find a sample connection configuration file localstorage.yaml in the $REPLICANT_HOME/conf/conn directory.

Specify the connection details in the following manner:

type: LOCALSTORAGE
access-method: LOCAL
storage-location: "PATH_TO_STORAGE_LOCATION"
file-format: CSV

Replace PATH_TO_STORAGE_LOCATION with the location on your disk where you want to save your data.

II. Set up Applier configuration #

To configure replication mode according to your requirements, specify your configuration in the Applier configuration file. You can find a sample Applier configuration file localstorage.yaml in the $REPLICANT_HOME/conf/dst directory.

Common configuration parameters #

encode-binary-to-base64

{true|false}.

Whether to convert binary data into base64 when writing to local storage.

Default: false.

Configure snapshot mode #

For operating in snapshot mode, specify your configuration under the snapshot section of the configuration file:

snapshot:
  threads: 16

  txn-size-rows: 1_000_000
  max-file-size: 33_554_432 # 32MB

  delimiter: ','
  quote: '"'
  escape: '"'

  include-header: {true|false}

  zip: GZIP

  encode-binary-to-base64: {true|false}

The include-header parameter allows you to choose whether or not you want CSV header in CSV file.

For more information about the Applier parameters for snapshot mode, see Snapshot mode.

Configure real-time replication #

For operating in realtime mode, specify your configuration under the realtime section of the conifiguration file:

realtime:
  threads: 16
  encode-binary-to-base64 : true

For more information about the configuration parameters for realtime mode, see Realtime mode.