This video will explain how to setup a data flow job that moves data from pub/sub topic to Big Query table.
Commands used:
gcloud pubsub topics create MyTopic01
gsutil mb gs://dataengineer-01
bq mk mydataset01
bq mk mydataset01.mytable01 name:STRING,country:STRING
Message format
{
"name" : "John",
"country" : "US"
}
Негізгі бет 01 Cloud Dataflow - Pub/Sub to Big Query Streaming
Пікірлер: 37