A Structured Streaming job deployed to databricks video
data-engineer-professional video for a Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present,
Answer
Full Certification Question
A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds. Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?