[Enhancement] Reduce txn log writes and deletes #42542
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sduzh
force-pushed
the
enhancement/merge-txn-log
branch
from
46c11e2 to
958f194
Compare
23 tasks
This was referenced Mar 15, 2024
|
@Mergifyio reabase |
❌ Sorry but I didn't understand the command. Please consult the commands documentation 📚. |
|
@Mergifyio rebase |
✅ Branch has been successfully rebased |
sduzh
force-pushed
the
enhancement/merge-txn-log
branch
from
6584800 to
be87f77
Compare
|
@wyb PTAL |
sduzh
force-pushed
the
enhancement/merge-txn-log
branch
3 times, most recently
from
40dd029 to
3f69af3
Compare
sduzh
force-pushed
the
enhancement/merge-txn-log
branch
4 times, most recently
from
79a16c2 to
319a221
Compare
kevincai
previously approved these changes
Apr 16, 2024
|
@Mergifyio rebase |
✅ Branch has been successfully rebased |
sduzh
force-pushed
the
enhancement/merge-txn-log
branch
from
35b9a12 to
091d38a
Compare
Signed-off-by: Alex Zhu <[email protected]>
Signed-off-by: Alex Zhu <[email protected]>
sduzh
force-pushed
the
enhancement/merge-txn-log
branch
from
091d38a to
0a10749
Compare
kevincai
approved these changes
Apr 19, 2024
|
wyb
approved these changes
Apr 19, 2024
Seaven
approved these changes
Apr 19, 2024
HangyuanLiu
approved these changes
Apr 19, 2024
[FE Incremental Coverage Report]✅ pass : 87 / 105 (82.86%) file detail
|
[BE Incremental Coverage Report]✅ pass : 313 / 357 (87.68%) file detail
|
24 tasks
imay
pushed a commit
that referenced
this pull request
May 8, 2024
Why I'm doing: In order to be able to reduce reads and writes to the object store, we added COMBINED TXN LOG in #42542, i.e., only one txn log file is written per partition instead of one per Tablet. However, in #42542, only stream load and routine load support combined txn log, the common insert into and broker load can not use combined txn log. What I'm doing: Support combined txn log for insert and broker load. Avoid sending invalid txn log deletion requests when combined txn log and batch publish are turned on at the same time Signed-off-by: Alex Zhu <[email protected]>
node
pushed a commit
to vivo/starrocks
that referenced
this pull request
May 9, 2024
Why I'm doing: In order to be able to reduce reads and writes to the object store, we added COMBINED TXN LOG in StarRocks#42542, i.e., only one txn log file is written per partition instead of one per Tablet. However, in StarRocks#42542, only stream load and routine load support combined txn log, the common insert into and broker load can not use combined txn log. What I'm doing: Support combined txn log for insert and broker load. Avoid sending invalid txn log deletion requests when combined txn log and batch publish are turned on at the same time Signed-off-by: Alex Zhu <[email protected]>
24 tasks
chaoyli
pushed a commit
that referenced
this pull request
Nov 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
In the current implementation, for each data load, each Tablet creates a txn log file on the object store and deletes it after a successful publish version. This will bring a lot of object store access, which can easily lead to request limit exceeded problem, and also some API access billing.
What I'm doing:
Instead of creating a txn log file for each Tablet at the end of data loading, the OlapTableSink node creates a single combined txn log file containing all tablet txn logs. During the publish version phase, each tablet will get the combined txn log file and find its own txn log from it.
TODO:
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: