Limit concurrent ObjectStore access to avoid resource limitations in constrained environments #2457
Assignees
Labels
enhancement
New feature or request
Comments
|
take |
rtyler
added a commit
that referenced
this issue
May 1, 2024
…jectStoreFactory (#2458) # Description This PR adds a configuration to control concurrent access to the underlying object store. It also includes a visibility change to the S3LogStoreFactory to align it with all other provider implementations. # Related Issue(s) - closes #2457 - resolves #2353 # Documentation https://docs.rs/object_store/latest/object_store/limit/struct.LimitStore.html --------- Co-authored-by: Michele Vigilante <[email protected]> Co-authored-by: R. Tyler Croy <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Add a configuration to limit concurrent object store access for delta operations. This can be accomplished using LimitStore which wraps an ObectStore to limit concurrency.
Use Case
When running ETLs utilizing deltatables on AWS lambda accessing S3 it is quite easy to run into the lambda file descriptor limit of 1024 if you're querying many tables. The issue surfaces as an error from hyper panicking with "Too many open files".
The file descriptor limit on AWS lambda is not configurable.
Related Issue(s)
#2353
The text was updated successfully, but these errors were encountered: