data-engineer video for a healthcare company is developing a serverless ETL pipeline to process medical records in compliance with healthcare regulations. The
A healthcare company is developing a serverless ETL pipeline to process medical records in compliance with healthcare regulations. The pipeline needs to perform the following steps: Data arrives in CSV format and is stored in an Amazon S3 bucket. AWS Lambda should automatically trigger when new files are uploaded. Lambda should transform the data into a standard JSON format and send it to Amazon Kinesis Data Streams for real-time processing. Processed data should be stored in Amazon DynamoDB for fast querying by the downstream applications. The solution should handle sudden increases in incoming data without manual intervention. Which solution will meet the requirements while maintaining a cost-effective, fully managed, and scalable architecture?