data-engineer video for an Internet-of-Things (IoT) devices company uses Amazon S3 as the data lake to store the input data that is ingested from the field
An Internet-of-Things (IoT) devices company uses Amazon S3 as the data lake to store the input data that is ingested from the field devices on an hourly basis. The ingested data has attributes such as the device type, ID of the device, the status of the device, the timestamp of the event, the source IP address, etc. The data runs into millions of records per day and the company wants to run complex analytical queries on this data on a daily basis for product improvements for each device type. Which is the most optimal way to save this data to get the best performance from the millions of data points saved on a daily basis?