Replication methods for BigQuery #
Load data with load job method #
The conventional load job method is the default method of loading data into BigQuery partitions and tables. See the sample configurations for snapshot and realtime to understand how to use the load job method.
Load data using the Storage Write API #
This method of replication uses the BigQuery Storage Write API to stream data into BigQuery.
- Make sure you have the necessary
bigquery.tables.updateDatapermissions. For more information, see Required permissions.
- Provide your credentials by setting the
GOOGLE_APPLICATION_CREDENTIALSenvironment variable to the path of the JSON file that contains your service account key. Setting this variable authenticates you so that Replicant can use the Storage Write API.
You can use the Storage Write API method by setting the
use-write-storage-api parameter to
true under the
snapshot section of the Applier configuration file. See the sample configurations for snapshot and realtime to understand how to use the Storage Write API.
Decide which method of replication to use #
The load job method is the conventional method of loading data, while the Storage Write API is the latest streaming API by BigQuery. When choosing between the load job method and the Storage Write API, please consider the advantatages of using the Storage Write API.