lance-datafusion as a subproject

[jackye1995]

Recently there are a lot of development and discussions around adding UDFs, UDTFs, and views to work with Lance dataset for SQL usage. And when we say SQL, it's basically the DataFusion SQL exposed through Lance interface.

However, doing it at the dataset level seems to be problematic, it is hard to get the ideal behavior of SQL because a single dataset do not have the vision of everything going on with other datasets. It imposes challenges in when to register a table, what name to use to register the table, how to avoid duplications, how to register UDTF for all tables, how to create views with deterministic table reference, etc.

We have discussed about this a few times:

• contains_token UDT: feat: add contains_tokens udf #4420
• FTS UDFT: feat: add udtf for fts query #4684
• View: Support views on lance dataset #4733
• Single session context for SQL: feat: use single session context in dataset sql #4464

Proposal: a lot of these discussions come from the fact that people want to use Lance with a full SQL experience, and DataFusion is at the best position to be the de-facto SQL engine for Lance. In that case, I think we should properly create a lance-datafusion subproject that implements the CatalogProvider in datafusion using Lance namespace, just like what we have done in Spark and Ray.

What do we think?

cc @wjones127 @timsaucer @westonpace @wojiaodoubao @yanghua @jaystarshot

[timsaucer]

It's definitely an interesting topic. At Rerun I am currently working on an implementation of CatalogProvider that is similar to what you're discussing. Our use case is slightly different, I suspect, as we have other dataset-like objects we're also working to support. I do think there is likely a fair amount of overlap in goals/work.

Taking off my Rerun hat and putting on my DataFusion PMC hat, I think this is a great idea! We'd love to see more implementations of catalog providers, and especially working to make that a pleasant experience with datafusion-python would be a personal win.

What I don't know is what kind of support I can provide. I can share the code I've been putting together so far for the rerun implementation and I've been finding a couple of rough spots. I'd also be happy to get on a call (or use the DataFusion weekly community meeting, Wed 11am EST) to discuss.

[jackye1995]

we have other dataset-like objects we're also working to support.

do you mean view, MV, those types of objects?

[timsaucer]

No, we have our own file types that have many similarities to lance tables. Our CatalogProvider implementation will need to handle both our files and lance tables. So it's a bit of a different focus but many of the needs will overlap.

[wjones127]

I've recently been thinking about something that parallels this: I'd like to separate the Dataset implementation into two levels:

1. A core engine-agnostic implementation. This would contain things like Manifest, Transaction, CommitBuilder, IndexMetadata, write_fragments.
2. A DataFusion later that implements operations. This would contain things like Update, Delete, MergeInsert and queries.

The advantages of this would be:

1. It would make the core lance package much more focused on the table format itself. I think the limited scope would help us figure out a stable 1.0 version of the format faster.
2. It would allow users that don't care about DataFusion to not have to compile it as a dependency. For example, I don't think there's a reason for the lance-spark similar to require this.

Do you think it would be appropriate to combine (2) with your current idea of lance-datafusion?

IMO, I think it would be nice to have the datafusion-lance implementation in the main Lance repo. Part of this is because I think we will want to include it in the Python package for the foreseeable future.

[westonpace]

I think I agree with this approach but why stop at two layers 😆

• lance-table - Pull more of lance into lance-table so that lance-table is able to commit operations and handle conflict resolution. It would be nice if this layer didn't even have a notion of a compute engine. As it stands today the existing lance-table crate is a little anemic and not much more than protobuf wrappers so I think it makes sense to move more functionality here.
• lance-dataset - Define an abstraction for a lance compute engine. Move most of Dataset's functionality on top of this abstraction. Define a Dataset which combines a compute engine and lance-table.
• lance-datafusion-engine - A reference implementation of the compute engine based on datafusion (i.e. what we have today).
• lance-datafusion-provider - A datafusion table provider that sits on top of lance-dataset but not lance-datafusion-engine (this is confusing but imagine a user wants to do user-app->datafusion->lance-dataset->spark-engine)

[wjones127]

You're totally right that we could put the core engine-agnostic stuff in lance-table.

[westonpace]

I think views and UDFs are even a separate discussion.

Compute functionality (e.g. UDFs) for multi-media types should probably be in standalone projects that only depend on Arrow. They can have a datafusion feature that adds the shims needed to register them into Datafusion.

Views are pretty high up the stack and I'm not sure I understand enough yet to really define anything.

[wojiaodoubao]

I like this great design! Allow me to explain my understanding of the new design. It separates the Lance & DataFusion relationship into two parts:

• SQL experience: The lance-datafusion-provider crate, which uses DataFusion as the SQL query engine.
• Dataset search: The lance-datafusion-engine crate, which implements Lance's query logic using DataFusion.

At the same time, it abstracts out the engine-agnostic lance-table crate, which focuses on the table format.

I would like to use two examples to illustrate my understanding of the new design:

• Dataset::scan:
  • It first retrieves fragments and indexs through LanceTable,
  • Then passes the fragments, index, and search conditions to the SearchEngine to complete the query.
  • Fragments and Index are abstractions in the lance-table crate, which allows SearchEngine to avoid directly depending on Dataset.
• Dataset::delete:
  • It first scans the dataset using Dataset::scan to construct DeleteData,
  • Then submits the DeleteData through LanceTable to complete the deletion.

I thought of some points that might need attention:

• Index dependency on UDFs: Currently, the JSON index depends on the JSON UDF, which makes it difficult to place the JSON UDF in the lance-datafusion-provider.
• UDF pushdown: In the new design, most UDF implementations can be placed in the lance-datafusion-provider crate, but their pushdown logic still needs to be implemented in the lance-datafusion-engine crate.