feat(python): add filters argument to DeltaTable.to_pandas() for filter pushdown

[ognis1205]

Description

This pull request adds support for filtering to the to_pandas method of DeltaTable. The new filters argument allows
users to specify filtering criteria in a format that is compatible with pyarrow.compute.Expression.

• Adding the filters argument to DeltaTable.to_pandas.
• Adding the _filters_to_expression function to table module, which is based on this implementation, but with improved type consistency.
• Based on the existing conventional unit tests, I did not add any additional test cases for this feature. Instead, I tested the feature on my local development environment.
• Adding a unit test for the feature.

Related Issue(s)

• closes feat(python): add filters argument to DeltaTable.to_pandas() for filter pushdown #1316

Documentation

The DNF filter format.

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[ognis1205]

https://github.com/delta-io/delta-rs/actions/runs/4926020116/jobs/8800992032?pr=1349#step:8:18

I will implement the fallback function of pyarrow.parquet.core.filters_to_expression for PyArrow 7.0.0.
I will add _filters_to_expression method.

[ognis1205]

https://github.com/delta-io/delta-rs/actions/runs/4943095151/jobs/8837254340?pr=1349#step:5:545

I think the above error is not relevant to this PR.

Code example:

import deltalake

dt = deltalake.DeltaTable("../../rust/tests/data/COVID-19_NYT")
# "date > 2021-02-20" OR "state in ["Alabama", "Wyoming"]"
df = dt.to_pandas(filters=[[("date", ">", "2021-02-20")], [("state", "in", ["Alabama", "Wyoming"])]])
# "date > 2021-02-20" AND "state in ["Alabama", "Wyoming"]"
df = dt.to_pandas(filters=[("date", ">", "2021-02-20"), ("state", "in", ["Alabama", "Wyoming"])])

[wjones127]

I'd still like you to add a test at least for to_pyarrow_table() exercising the new argument. Ideally it should include a case for a conjunctionm one with a disjunction, and one single predicate.

[ognis1205]

@wjones127

Sorry I had forgotten to mention you, but I believe I have now resolved all the issues. Would it be possible for you to review the PR again when you have a moment? I would really appreciate it. Thank you!

[wjones127]

Thanks, those tests look great!

[wjones127]

Thanks, those tests look great!