This is a dedicated watch page for a single video.
A data engineering team is working on a complex pipeline with trillions of rows in each table. They decide to persist in some of the frequently used DataFrames to fasten the processing of the queries. One of the data engineers ran the persist() command over the DataFrame and immediately checked the Spark UI’s Storage Tab but was unable to find the information about the persisted DataFrame. What could be the possible reason?