[Feature] Support pk index major compaction in cloud native table

[TszKitLo40]

Why I'm doing:

PK index major compaction is not supported in cloud native table which will cause write amplification.

What I'm doing:

Support pk index major compaction in cloud native table.
WIthout pk index major compaction

	publish cost(ms)	MAX IO(MB/s)
1G + insert + 1/100	121442	54
1G + upsert + 1/100	291326	73
1G + upsert + 1/10000	338894	46.7
2.5G + insert + 1/100	710491	121
2.5G + upsert + 1/100	1419680	121
2.5G + upsert + 1/10000	307463	121

With pk index major compaction

	publish cost(ms)	MAX IO(MB/s)
1G + insert + 1/100	52866	64.1
1G + upsert + 1/100	125956	80.8
1G + upsert + 1/10000	405559	46.7
2.5G + insert + 1/100	191436	117
2.5G + upsert + 1/100	389931	108
2.5G + upsert + 1/10000	223999	114

In trace log, we can see that the time of primary_index_commit_latency_us is much less that without major compaction. Because merge compaction is not needed in commit.

I0301 13:51:28.410473  3076 lake_service.cpp:237] Published txns=5050. tablets=10163 cost=1869193us, trace: {"child_traces":[["PublishTablet",{"base_version":93,"deletes":0,"do_update_latency_us":924748,"new_del":0,"primary_index_commit_latency_us":909008,"primary_index_load_latency_us":0,"queuing_latency_us":20,"rewrite_segment_latency_us":19,"rowsetid":93,"state_bytes":9205820,"tablet_id":10163,"total_del":0,"update_index_latency_us":924788,"upsert_rows":151201,"upserts":1}]]}

I0301 13:52:56.940850  4938 lake_service.cpp:237] Published txns=5031. tablets=10139 cost=11237925us, trace: {"child_traces":[["PublishTablet",{"base_version":102,"deletes":0,"do_update_latency_us":870584,"new_del":0,"primary_index_commit_latency_us":10342835,"primary_index_load_latency_us":0,"queuing_latency_us":16,"rewrite_segment_latency_us":14,"rowsetid":102,"state_bytes":10569270,"tablet_id":10139,"total_del":0,"update_index_latency_us":870614,"upsert_rows":151124,"upserts":1}]]}

Fix #45740

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.3
  • 3.2
  • 3.1
  • 3.0
  • 2.5

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 112 / 132 (84.85%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/storage/tablet_manager.cpp	0	1	00.00%	[769]
🔵	be/src/storage/lake/local_pk_index_manager.cpp	44	61	72.13%	[210, 211, 216, 217, 221, 222, 232, 233, 234, 247, 248, 261, 262, 263, 267, 275, 283]
🔵	be/src/storage/persistent_index_compaction_manager.cpp	16	18	88.89%	[52, 81]
🔵	be/src/storage/lake/local_pk_index_manager.h	1	1	100.00%	[]
🔵	be/src/storage/olap_server.cpp	4	4	100.00%	[]
🔵	be/src/storage/lake/lake_primary_index.h	4	4	100.00%	[]
🔵	be/src/storage/persistent_index.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/lake_primary_index.cpp	13	13	100.00%	[]
🔵	be/src/storage/lake/update_manager.cpp	13	13	100.00%	[]
🔵	be/src/storage/lake/txn_log_applier.cpp	1	1	100.00%	[]
🔵	be/src/storage/tablet_updates.h	1	1	100.00%	[]
🔵	be/src/storage/storage_engine.cpp	2	2	100.00%	[]
🔵	be/src/storage/persistent_index_tablet_loader.cpp	3	3	100.00%	[]
🔵	be/src/storage/lake/lake_local_persistent_index.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/lake_local_persistent_index.h	2	2	100.00%	[]
🔵	be/src/storage/lake/lake_local_persistent_index_tablet_loader.h	2	2	100.00%	[]
🔵	be/src/storage/lake/lake_local_persistent_index_tablet_loader.cpp	4	4	100.00%	[]

[github-actions]

@Mergifyio backport branch-3.2

[mergify]

backport branch-3.2

✅ Backports have been created

• #46000 [Feature] Support pk index major compaction in cloud native table (backport #41737) has been created for branch branch-3.2 but encountered conflicts

[TszKitLo40]

https://github.com/Mergifyio backport branch-3.3

[mergify]

backport branch-3.3

✅ Backports have been created

• #46002 [Feature] Support pk index major compaction in cloud native table (backport #41737) has been created for branch branch-3.3