Skip to content

Commit fd0e6f3

Browse files
davidcavazosdavidcavazos
authored
Update commands to GA (GoogleCloudPlatform#3683)
1 parent cc59996 commit fd0e6f3

File tree

4 files changed

+39
-115
lines changed

4 files changed

+39
-115
lines changed

‎dataflow/flex-templates/kafka_to_bigquery/Dockerfile

-24
This file was deleted.

‎dataflow/flex-templates/kafka_to_bigquery/README.md

+16-12
Original file line numberDiff line numberDiff line change
@@ -223,34 +223,38 @@ gcloud compute instances create-with-container kafka-vm \
223223
First, let's build the container image.
224224
225225
```sh
226-
export TEMPLATE_IMAGE="gcr.io/$PROJECT/samples/dataflow/streaming-beam-sql:latest"
227-
228226
# Build and package the application as an uber-jar file.
229227
mvn clean package
230-
231-
# Build the Dataflow Flex template image into Container Registry.
232-
gcloud builds submit --tag "$TEMPLATE_IMAGE" .
233228
```
234229
235230
Now we can create the template file.
236231
237232
```sh
233+
export TEMPLATE_IMAGE="gcr.io/$PROJECT/samples/dataflow/kafka-to-bigquery-sql:latest"
238234
export TEMPLATE_PATH="gs://$BUCKET/samples/dataflow/templates/kafka-to-bigquery.json"
239235
240236
# Build the Flex Template.
241-
gcloud beta dataflow flex-template build $TEMPLATE_PATH \
242-
--image "$TEMPLATE_IMAGE" \
243-
--sdk-language "JAVA" \
244-
--metadata-file "metadata.json"
237+
gcloud dataflow flex-template build $TEMPLATE_PATH \
238+
--image-gcr-path "$TEMPLATE_IMAGE" \
239+
--sdk-language "JAVA" \
240+
--flex-template-base-image JAVA11 \
241+
--metadata-file "metadata.json" \
242+
--jar "target/kafka-to-bigquery-1.0.jar" \
243+
--env FLEX_TEMPLATE_JAVA_MAIN_CLASS="org.apache.beam.samples.KafkaToBigQuery"
245244
```
246245
247246
Finally, to run a Dataflow job using the template.
248247
249248
```sh
249+
export REGION="us-central1"
250+
250251
# Run the Flex Template.
251-
gcloud beta dataflow flex-template run "kafka-to-bigquery-`date +%Y%m%d-%H%M%S`" \
252-
--template-file-gcs-location "$TEMPLATE_PATH" \
253-
--parameters "inputTopic=messages,outputTable=$PROJECT:$DATASET.$TABLE,bootstrapServer=$KAFKA_ADDRESS:9092"
252+
gcloud dataflow flex-template run "kafka-to-bigquery-`date +%Y%m%d-%H%M%S`" \
253+
--template-file-gcs-location "$TEMPLATE_PATH" \
254+
--parameters inputTopic="messages" \
255+
--parameters outputTable="$PROJECT:$DATASET.$TABLE" \
256+
--parameters bootstrapServer="$KAFKA_ADDRESS:9092" \
257+
--region "$REGION"
254258
```
255259
256260
Run the following query to check the results in BigQuery.

‎dataflow/flex-templates/streaming_beam_sql/Dockerfile

-24
This file was deleted.

‎dataflow/flex-templates/streaming_beam_sql/README.md

+23-55
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ to transform the message data, and writes the results to a
105105
* [pom.xml](pom.xml)
106106
* [metadata.json](metadata.json)
107107

108-
### Building a container image
108+
### Build the Flex Template
109109

110110
> <details><summary>
111111
> <i>(Optional)</i> Run the Apache Beam pipeline locally for development.
@@ -139,55 +139,11 @@ This *Uber JAR* file has all the dependencies embedded so it.
139139
You can run this file as a standalone application with no external
140140
dependencies on other libraries.
141141

142-
Now, we build the
143-
[Docker](https://docs.docker.com/engine/docker-overview/)
144-
image for the Apache Beam pipeline.
145-
We are using
146-
[Cloud Build](https://cloud.google.com/cloud-build)
147-
so we don't need a local installation of Docker.
148-
149-
> ℹ️ You can speed up subsequent builds with
150-
> [Kaniko cache](https://cloud.google.com/cloud-build/docs/kaniko-cache)
151-
> in Cloud Build.
152-
>
153-
> ```sh
154-
> # (Optional) Enable to use Kaniko cache by default.
155-
> gcloud config set builds/use_kaniko True
156-
> ```
157-
158-
Cloud Build allows you to
159-
[build a Docker image using a `Dockerfile`](https://cloud.google.com/cloud-build/docs/quickstart-docker#build_using_dockerfile).
160-
and saves it into
161-
[Container Registry](https://cloud.google.com/container-registry/),
162-
where the image is accessible to other Google Cloud products.
163-
164-
```sh
165-
export TEMPLATE_IMAGE="gcr.io/$PROJECT/samples/dataflow/streaming-beam-sql:latest"
166-
167-
# Build the image into Container Registry, this is roughly equivalent to:
168-
# gcloud auth configure-docker
169-
# docker image build -t $TEMPLATE_IMAGE .
170-
# docker push $TEMPLATE_IMAGE
171-
gcloud builds submit --tag "$TEMPLATE_IMAGE" .
172-
```
173-
174-
> ℹ️ We use the [`.gcloudignore`](.gcloudignore) file to ignore large
175-
> files not used for the container image, such as build files.
176-
> This helps speed up the build by uploading less data.
177-
>
178-
> To learn more about `.gcloudignore`, see
179-
> [`gcloud topic gcloudignore`](https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore)
180-
181-
Images starting with `gcr.io/PROJECT/` are saved into your project's
182-
Container Registry, where the image is accessible to other Google Cloud products.
183-
184-
### Creating a Flex Template
185-
186142
To run a template, you need to create a *template spec* file containing all the
187143
necessary information to run the job, such as the SDK information and metadata.
188144

189145
The [`metadata.json`](metadata.json) file contains additional information for
190-
the template such as the "name", "description", and input "parameters" field.
146+
the template such as the `name`, `description`, and input `parameters` field.
191147

192148
We used
193149
[regular expressions](https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-language-quick-reference)
@@ -198,31 +154,43 @@ and [BigQuery table](https://cloud.google.com/bigquery/docs/tables#table_naming)
198154
The template file must be created in a Cloud Storage location,
199155
and is used to run a new Dataflow job.
200156

157+
A container image is created, which includes a self-contained application of your pipeline.
158+
Images starting with `gcr.io/PROJECT/` are saved into your project's
159+
Container Registry, where the image is accessible to other Google Cloud products.
160+
201161
```sh
202162
export TEMPLATE_PATH="gs://$BUCKET/samples/dataflow/templates/streaming-beam-sql.json"
163+
export TEMPLATE_IMAGE="gcr.io/$PROJECT/samples/dataflow/streaming-beam-sql:latest"
203164
204165
# Build the Flex Template.
205-
gcloud beta dataflow flex-template build $TEMPLATE_PATH \
206-
--image "$TEMPLATE_IMAGE" \
207-
--sdk-language "JAVA" \
208-
--metadata-file "metadata.json"
166+
gcloud dataflow flex-template build $TEMPLATE_PATH \
167+
--image-gcr-path "$TEMPLATE_IMAGE" \
168+
--sdk-language "JAVA" \
169+
--flex-template-base-image JAVA11 \
170+
--metadata-file "metadata.json" \
171+
--jar "target/streaming-beam-sql-1.0.jar" \
172+
--env FLEX_TEMPLATE_JAVA_MAIN_CLASS="org.apache.beam.samples.StreamingBeamSQL"
209173
```
210174
211175
The template is now available through the template file in the Cloud Storage
212176
location that you specified.
213177
214-
### Running a Dataflow Flex Template pipeline
178+
### Running a Flex Template pipeline
215179
216180
You can now run the Apache Beam pipeline in Dataflow by referring to the
217181
template file and passing the template
218182
[parameters](https://cloud.devsite.corp.google.com/dataflow/docs/guides/specifying-exec-params#setting-other-cloud-dataflow-pipeline-options)
219183
required by the pipeline.
220184
221185
```sh
222-
# Run the Flex Template.
223-
gcloud beta dataflow flex-template run "streaming-beam-sql-`date +%Y%m%d-%H%M%S`" \
224-
--template-file-gcs-location "$TEMPLATE_PATH" \
225-
--parameters "inputSubscription=$SUBSCRIPTION,outputTable=$PROJECT:$DATASET.$TABLE"
186+
export REGION="us-central1"
187+
188+
# Run the template.
189+
gcloud dataflow flex-template run "streaming-beam-sql-`date +%Y%m%d-%H%M%S`" \
190+
--template-file-gcs-location "$TEMPLATE_PATH" \
191+
--parameters inputSubscription="$SUBSCRIPTION" \
192+
--parameters outputTable="$PROJECT:$DATASET.$TABLE" \
193+
--region "$REGION"
226194
```
227195
228196
Check the results in BigQuery by running the following query:

0 commit comments

Comments
 (0)