Workflows · From Tasks · from_bigquery

Purpose

Imports data from Google BigQuery. If you have an Google Cloud configured as your client_cloud you should probably use the do task.

Method of use

Before you can download data from Google BigQuery you have to give Workflows acces. Take the steps below to download data with a service account.

Create a service account or re-use a service account of a Google Cloud environment.
Give the service account BigQuery Editor permissions to the BigQuery datasets.
Have the service account added to the Workflows connections by a Onesecondbefore staff member
If you use Google Cloud, you're done. If you use another cloud, add the connection string to conn_id

Configuration

Example usage

extract:
    query: SELECT 1 AS int_field, '{{ task.run_id }}' AS string_field, CURRENT_TIMESTAMP() AS timestamp_field
    db_conn_id: google_cloud_default
    storage_conn_id: google_cloud_default
    project_id: my-project-id
    dataset_id: my-dataset-id-for-temp-table
    bucket: bucket-to-extract-tmp-files-to
    folder: folder-in-bucket-to-extract-tmp-files-to

Properties

property	type	mandatory	description
`query`	string	no	Use either query or template (below). Query to be executed, whose results will be uploaded to the destination.
`template`	string	no	Use either query(above) or template. Contains a link to a file in the `includes` folder in your repository that contains the SQL statement. This query will be executed and the results will be uploaded to the destination.
`db_conn_id`	string	no	Connection to use to connect to the BigQuery database
`db_location`	enumerator(bigquery query locations)	no	Query location. Default is EU. Currently only supported in Google BigQuery.
`storage_conn_id`	string	no	Connection to use to connect to Google Cloud Storage.
`project_id`	string	no	Project ID for table extract to Google Cloud Storage.
`dataset_id`	string	no	Dataset ID in which the temporary table will be created. The table id will be `_tmp_{{ task.run_id }}`
`bucket`	string	no	Bucket to which the data will be extracted from the temporary table.
`folder`	string	no	Folder to which the data will be extracted from the temporary table.

Details

item	description
`API`	BigQuery API
`Pre-formatted schema`	BigQuery uses the schema of the query. Use config schema if you want to add descriptions.