Replication methods for BigQuery #

For realtime and snapshot replication into BigQuery, Replicant supports the following two methods:

• Loading data with conventional load job method.
• Streaming data using BigQuery Storage Write API.

Load data with load job method #

The conventional load job method is the default method of loading data into BigQuery partitions and tables. See the sample configurations for snapshot and realtime to understand how to use the load job method.

Load data using the Storage Write API #

This method of replication uses the BigQuery Storage Write API to stream data into BigQuery.

Requirements #

• Make sure you have the necessary bigquery.tables.updateData permissions. For more information, see Required permissions.
• Provide your credentials by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of the JSON file that contains your service account key. Setting this variable authenticates you so that Replicant can use the Storage Write API.

Configuration #

You can use the Storage Write API method by setting the use-write-storage-api parameter to true under the realtime or snapshot section of the Applier configuration file. See the sample configurations for snapshot and realtime to understand how to use the Storage Write API.

Decide which method of replication to use #

The load job method is the conventional method of loading data, while the Storage Write API is the latest streaming API by BigQuery. When choosing between the load job method and the Storage Write API, please consider the advantatages of using the Storage Write API.