Scenario: You need to schedule sequential load and transformation jobs for hundreds of tables. Data files will be added to a Cloud Storage bucket without a fixed schedule. Once the data is added, a Dataproc job should perform transformations and write the data to BigQuery. Afterward, you need to run different transformation jobs in BigQuery, which might take hours to complete. You need to determine the most efficient and maintainable workflow to process all tables and provide the freshest data to end users. Question: What is the best approach to efficiently and maintainable process the data and provide the freshest data to end users?