Low-level transaction "write" API(s)

[rtyler]

In a video discussion with @xianwill and @houqp, they were discussing the gists of what a low-level transaction API might look like for the Rust crate.

I wanted to open that discussion up a bit further, but also try out this new GitHub feature 😄

Below is my interpretation of what they were saying with some personal opinion thrown in:

The underlying parquet crate doesn't yet support writes, but at a low-level the deltalake crate should provide write-level primitives to the transaction log. For example, if I already have a parquet file created by something else, Is should be able to provide a path/URI (not sure which) in a deltatable.add_file type API, which creates a new transaction and gives me the transaction details back.

@houqp mentioned reviewing the delta protocol for some of the specific semantic needed to be implemented here.

Perhaps one approach to continue this discussion would be for @xianwill to share some sketches of what these low-level API additions might look like.

[houqp]

The only thing i would add to what Tyler said is we will probably need to wrap the various low level write primitive actions with transaction apis, for example:

let mut txn = table.new_transaction()
txn.add_file(vec!["a.parquet", "b.parquet", "c.parquet"])
txn.remove_file(...)
match txn.commit() {
    Ok(()) => ,
    Err(e) => {
        // handle commit conflicts and other errors
    }
}

We could also use https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/DeltaLog.scala for some inspirations.

[rtyler]

Some notes from our open development meeting

• Expect Paths relative to the table itself, absolute paths are not valid in the transaction log
• First iteration with the local file system rather than targeting the cloud storage providers
• Transaction commit is where we'd try to do the optimistic write
• transaction writer expects that the paths already exist on disk. Basically the parquet file should already be present in storage, and then the transaction APIs only add
• Stale/outstanding pull request in Arrow for doing some writes with arbitrary bytes ARROW-8796: [Rust] feat: Allow writers to use &mut Vec apache/arrow#7176
  • @xianwill was hoping for an API which took a struct implementing a Write trait.
  • Challenging constraints to solve from @nevi-me
    • Have to support try_clone
    • @nevi-me still needs to do some work on that pull request to extend it

[xianwill]

I have started an early draft to work through some of the transaction API concepts and get more familiar with delta-rs. I haven't sketched the user facing transaction APIs yet. I wanted to lay down a foundation first to verify some of the underlying plumbing requirements.

Feel free to checkout the wip. I'll start laying sketches on top of this soon.

I'm using the StorageBackend api for now for writing logs. This may not be possible in the end. Just using it as a convenience.

[rtyler]

@xianwill now has #84 propsed, and I have no idea how to close this discussion. 😒

[xianwill]

Heh. Well good (for now at least)! This particular discussion thread should probably include a couple of additional PRs besides that one anyway.

[xianwill]

There are some additional follow-ups to this writer discussion beyond #84 that warrant further conversation.

This particular one probably doesn't need a ton of discussion cuz I definitely get the idea and agree with the goal. Either way, we still need to design and implement the action holder approach suggested by @nevi-me and corroborated by @houqp. For my own tastes, I think I prefer keeping a factory function outside of the transaction itself for creating actions. The transaction itself does not know any details really that should be included in the action and therefore cannot really limit the number parameters required for creating the action from internal state (ineffectual helper - spirit-wise somewhat similar to @rtyler's points about the create_transaction helper on DeltaTable). So, I think the API I would like to see looks something like:

let mut txn = table.create_transaction(...)

let my_add = Actions::new_add(...)
txn.add_action(my_add)

...

So basically, hang factory functions off of the Actions module and create an API on the DeltaTransaction to append these to an encapsulated vec of actions that should be committed.

[xianwill]

Another item to discuss is the transaction log storage backend for S3. We need "fail if file exists already" semantics for S3 to run the optimistic concurrency loop w/ an s3 storage backend. I'm thinking a dynamodb guard should work well, but, in these tumultuous times (lol) of rusoto maintenance mode, I feel a little gun shy about going all in on implementation.

[houqp]

Given that this is already implemented, I recommend us moving the S3 log storage discussion to a new thread. I am guessing locking discussion is the equivalent of closing an issue?

[rtyler]

Locking this discussion since we do have some preliminary code merged at this point in #248