Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Reduce parallelism when vacuuming dropped tables #16312

Merged
merged 1 commit into from
Aug 22, 2024

Conversation

dantengsky
Copy link
Member

@dantengsky dantengsky commented Aug 22, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Limit parallelism to a maximum of 3 when vacuuming dropped tables.

  • for each vacuum task, the tables passed to it will be processed sequentially
  • while removing one table's data, at most 1000 objects will be deleted (in batch)
  • let's assume that the rate limit is 3500 (individual) objects per second:
    A parallelism degree of up to 3 appears to be safe.

fixes: #16311

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - tweak vacuuming dropped table parallelism

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Aug 22, 2024
@dantengsky dantengsky force-pushed the refact-reduce-del-parallel branch from 16d0984 to bb96c84 Compare August 22, 2024 11:08
@dantengsky dantengsky marked this pull request as ready for review August 22, 2024 13:19
@dantengsky dantengsky added this pull request to the merge queue Aug 22, 2024
Merged via the queue into databendlabs:main with commit d6d2159 Aug 22, 2024
72 checks passed
@dantengsky dantengsky deleted the refact-reduce-del-parallel branch August 22, 2024 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-refactor this PR changes the code base without new features or bugfix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Reduce parallelism when vacuuming dropped tables
2 participants