Instead of spending all of this effort trying to optimize an OLAP data lake for an OLTP use case, why not ust use a regular OLTP database like Postgres or MySQL?
Even if the data size is "infinite" you can put it on PlanetScale or something like that. If the tables are well optimized with covering indexes, you can pull out thousands of rows in 100ms for point queries, which is plenty for "point" queries.
Pretty clever to keep the existing parquet files as is (assuming daily partitioned) to not break any analytical queries. Also creating this index for the first time would make me sweat. With exabytes in the lake, the backfill bill will be $$$ (even just scanning the key column).
I'm not sure about the AI agent angle though, why would it need that specific context on a user level? I guess they are exploring a "chat with my spotify history" or user generated playlists?
(also the article has several LLM smells. It's technically interesting but reads like slop at times)
Even if the data size is "infinite" you can put it on PlanetScale or something like that. If the tables are well optimized with covering indexes, you can pull out thousands of rows in 100ms for point queries, which is plenty for "point" queries.
I'm not sure about the AI agent angle though, why would it need that specific context on a user level? I guess they are exploring a "chat with my spotify history" or user generated playlists?
(also the article has several LLM smells. It's technically interesting but reads like slop at times)