WTF is an Analytics Lake: Building an Open Data Service Layer with Arrow, DuckDB and Semantic Layer

Have you implemented a bunch of data services interacting with both clients and each other? Has maintenance became a huge burden? Do you have to boiler-plate too often? Is the performance a concern?

In this workshop, we will present to you our project "FlexQuery" which can help you to find the light at the end of the tunnel.
With just a few lines of Python code, without boiler plating, you can build a new data service.
It fits extremely well into the whole ecosystem powered by Arrow and Arrow Flight.

Moreover, we provide a base set of data services for you:
- Data Source Connectors: Bridging data sources and your analytics environment seamlessly.
- Multi-Tier Cache Storage: High-performance cache storage spanning memory, disk, and object stores.
- Cache Enabled Query Execution: Real-time insights from cached data powered by DuckDB.
- Post-SQL Transformations: Precise data shaping with the versatility of Pandas or Polars.

Finally, we provide you a complete deployment flexibility - start really small and scale out, deploy anywhere (baremetal, VMs, K8S).

But we don't stop there.
Our ongoing work introduces a new service, proposing an optimal cache design based on physical statistics, our innovative semantic model, and user activity.
In the future, we plan to fine-tune a Large Language Model, and let it design the caches even better.

And we plan to make this open source for all to benefit. Join our workshop to learn more!