Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(base): rollback to kernel signal handler after dump backtrace #16828

Merged
merged 4 commits into from
Nov 13, 2024

Conversation

zhang2014
Copy link
Member

@zhang2014 zhang2014 commented Nov 13, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

chore(base): rollback to kernel signal handler after dump backtrace

  • add SIGABRT handler for catch std::process::abort()
  • avoid using head memory in signal handler
  • rollback to kernel default signal handler after dump backtrace

Suggestions from: @dantengsky

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@zhang2014 zhang2014 requested a review from dantengsky November 13, 2024 02:25
@github-actions github-actions bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Nov 13, 2024
@dantengsky
Copy link
Member

dantengsky commented Nov 13, 2024

maybe we could use SA_RESETHAND (and keep the signal_handler as it is, but the exit(1) should be removed), something like this:

-- diff is based on this PR

diff --git a/src/common/tracing/src/crash_hook.rs b/src/common/tracing/src/crash_hook.rs
index 8596d78576..5264d74424 100644
--- a/src/common/tracing/src/crash_hook.rs
+++ b/src/common/tracing/src/crash_hook.rs
@@ -254,20 +254,16 @@ unsafe extern "C" fn signal_handler(sig: i32, info: *mut libc::siginfo_t, uc: *m

     #[allow(unreachable_code)]
     if sig != libc::SIGTRAP {
-        match libc::SIG_ERR == libc::signal(sig, libc::SIG_DFL) {
-            true => std::process::exit(1),
-            false => match libc::raise(sig) {
-                0 => {}
-                _ => std::process::exit(1),
-            },
-        }
+        drop(guard);
+        let _ = std::io::stderr().flush();
     }
 }

 pub unsafe fn add_signal_handler(signals: Vec<i32>) {
     let mut sa = std::mem::zeroed::<libc::sigaction>();

-    sa.sa_flags = libc::SA_ONSTACK | libc::SA_SIGINFO;
+    sa.sa_flags = libc::SA_ONSTACK | libc::SA_SIGINFO | libc::SA_RESETHAND;
     sa.sa_sigaction = signal_handler as usize;

     libc::sigemptyset(&mut sa.sa_mask);

and tweak sa_flags later, when the the mechanism of dumping backtrace is stabilized (currently, the dumping thread may have no opportunity to receive and dump all the trace from pipe, IMHO)

update: std::process::exit(1) should be removed, let the default handler (which is reset to by using SA_RESETHAND) exit the process. the whole point is to ensure coredump could be generated (if backtrace could not be captured)

@zhang2014
Copy link
Member Author

refactored the signal handler to avoid using heap memory. please review it again.

@zhang2014 zhang2014 enabled auto-merge November 13, 2024 11:47
@zhang2014 zhang2014 added this pull request to the merge queue Nov 13, 2024
Merged via the queue into databendlabs:main with commit 6debfa2 Nov 13, 2024
74 of 75 checks passed
@zhang2014 zhang2014 deleted the chore/roll_back_signal_handler branch November 13, 2024 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-chore this PR only has small changes that no need to record, like coding styles.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants