Video upload date:  · Duration: PT1H46M27S  · Language: EN

You are operating a streaming Dataflow pipeline gcp video

data-engineer-pro video for you are operating a streaming Dataflow pipeline that ingests Pub/Sub messages triggered by Cloud Storage events. The pipeline is

This is a dedicated watch page for a single video.

Full Certification Question

You are operating a streaming Dataflow pipeline that ingests Pub/Sub messages triggered by Cloud Storage events. The pipeline is configured with Streaming Engine and Horizontal Autoscaling, and the maximum number of workers is set to 1000. One transformation in the pipeline reads a CSV file and emits one element per line. However, job performance is poor, and the pipeline consistently uses only 10 workers. The autoscaler is not adding more workers. What should you do to improve pipeline performance?