ClickHouse Stress Test (asan) for master
fatal_messages.txt Commit check_status.tsv clickhouse-server.err.log.zst clickhouse-server.final.log clickhouse-server.initial.log clickhouse-server.stress.log.zst coordination.tar dmesg.log gdb.log hung_check.log query_log.tsv.zst run.log stderr.log stdout.log stress_run_logs.tar.zst test_results.tsv trace_log.tsv.zst Task (github actions) Job (github actions)
| Test name | Test status |
|---|---|
| Hung check failed, possible deadlock found (see hung_check.log) | FAIL |
Using queries from \'/usr/share/clickhouse-test/queries\' directory Connecting to ClickHouse server...Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused Connection error, will retry: [Errno 111] Connection refused | |
| Fatal message in clickhouse-server.log (see fatal_messages.txt) | FAIL |
[ 2399 ] {b251e5f1-95f3-4d07-a0fc-6ae896a7e397} <Fatal> : Logical error: \'inserted\'.
[ 2367 ] {b251e5f1-95f3-4d07-a0fc-6ae896a7e397} <Fatal> : Logical error: \'inserted\'.
[ 1888 ] {} <Fatal> Application: Child process was terminated by signal 6.
[ 2399 ] {b251e5f1-95f3-4d07-a0fc-6ae896a7e397} <Fatal> : Logical error: \'inserted\'.
[ 2367 ] {b251e5f1-95f3-4d07-a0fc-6ae896a7e397} <Fatal> : Logical error: \'inserted\'.
[ 1888 ] {} <Fatal> Application: Child process was terminated by signal 6.
| |
| Found signal in gdb.log | FAIL |
2024-12-17 01:15:57 Thread 392 "LoadMarksThread" received signal SIGABRT, Aborted.
2024-12-17 01:15:57 [Switching to Thread 0x7f9da6fbb640 (LWP 2399)]
2024-12-17 01:15:57 0x00007fa0e20399fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
2024-12-17 01:15:57 #0 0x00007fa0e20399fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
2024-12-17 01:15:57 No symbol table info available.
2024-12-17 01:15:57 #1 0x00007fa0e1fe5476 in raise () from /lib/x86_64-linux-gnu/libc.so.6
2024-12-17 01:15:57 No symbol table info available.
2024-12-17 01:15:57 #2 0x00007fa0e1fcb7f3 in abort () from /lib/x86_64-linux-gnu/libc.so.6
2024-12-17 01:15:57 No symbol table info available.
2024-12-17 01:15:57 #3 0x000055d89bd72d08 in DB::abortOnFailedAssertion (description=...) at ./src/Common/Exception.cpp:50
2024-12-17 01:15:57 always_false = false
2024-12-17 01:15:57 #4 0x000055d8b09ac7be in DB::CleanupQueue::add (this=<optimized out>, key=...) at ./src/Interpreters/Cache/Metadata.cpp:376
2024-12-17 01:15:57 inserted = <optimized out>
2024-12-17 01:15:57 #5 0x000055d8b09a1a56 in DB::LockedKey::~LockedKey (this=0x603000e77360) at ./src/Interpreters/Cache/Metadata.cpp:677
2024-12-17 01:15:57 No locals.
2024-12-17 01:15:57 #6 0x000055d8b09b116d in std::__1::default_delete<DB::LockedKey>::operator()[abi:v15000](DB::LockedKey*) const (this=<optimized out>, __ptr=0x603000e77360) at ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48
2024-12-17 01:15:57 No locals.
2024-12-17 01:15:57 #7 std::__1::__shared_ptr_pointer<DB::LockedKey*, std::__1::default_delete<DB::LockedKey>, std::__1::allocator<DB::LockedKey> >::__on_zero_shared (this=<optimized out>) at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:263
2024-12-17 01:15:57 No locals.
2024-12-17 01:15:57 #8 0x000055d8b0977159 in std::__1::__shared_count::__release_shared[abi:v15000]() (this=0x6030014cb2a0) at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174
2024-12-17 01:15:57 No locals.
2024-12-17 01:15:57 #9 std::__1::__shared_weak_count::__release_shared[abi:v15000]() (this=0x6030014cb2a0) at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215
2024-12-17 01:15:57 No locals.
2024-12-17 01:15:57 #10 std::__1::shared_ptr<DB::LockedKey>::~shared_ptr[abi:v15000]() (this=0x7f9da583f1e0) at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702
2024-12-17 01:15:57 No locals.
2024-12-17 01:15:57 #11 DB::FileSegment::complete (this=<optimized out>) at ./src/Interpreters/Cache/FileSegment.cpp:718
2024-12-17 01:15:58 watch = {event = {t = 348}, watch = {start_ns = 2797306304126, stop_ns = 0, clock_type = 1, is_running = true}}
2024-12-17 01:15:58 locked_key = {__ptr_ = 0x603000e77360, __cntrl_ = 0x6030014cb2a0}
2024-12-17 01:15:58 segment_lock = {<std::__1::unique_lock<std::__1::mutex>> = {__m_ = 0x612002174ac8, __owns_ = true}, <No data fields>}
2024-12-17 01:15:58 is_downloader = <optimized out>
2024-12-17 01:15:58 is_last_holder = <optimized out>
2024-12-17 01:15:58 current_downloaded_size = <optimized out>
2024-12-17 01:15:58 scope_exit616 = <optimized out>
2024-12-17 01:15:58 #12 0x000055d8b097d84f in DB::FileSegmentsHolder::completeAndPopFrontImpl (this=0x6030012babe0) at ./src/Interpreters/Cache/FileSegment.cpp:921
2024-12-17 01:15:58 No locals.
2024-12-17 01:15:58 #13 DB::FileSegmentsHolder::~FileSegmentsHolder (this=<optimized out>) at ./src/Interpreters/Cache/FileSegment.cpp:933
2024-12-17 01:15:58 file_segment_it = <optimized out>
2024-12-17 01:15:58 watch = {event = {t = 353}, watch = {start_ns = 2797306303454, stop_ns = 0, clock_type = 1, is_running = true}}
2024-12-17 01:15:58 #14 0x000055d8ac4e4152 in std::__1::default_delete<DB::FileSegmentsHolder>::operator()[abi:v15000](DB::FileSegmentsHolder*) const (this=<optimized out>, __ptr=0x6030012babe0) at ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48
2024-12-17 01:15:58 No locals.
2024-12-17 01:15:58 #15 std::__1::unique_ptr<DB::FileSegmentsHolder, std::__1::default_delete<DB::FileSegmentsHolder> >::reset[abi:v15000](DB::FileSegmentsHolder*) (this=<optimized out>, __p=0x0) at ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305
2024-12-17 01:15:58 __tmp = 0x6030012babe0
2024-12-17 01:15:58 #16 std::__1::unique_ptr<DB::FileSegmentsHolder, std::__1::default_delete<DB::FileSegmentsHolder> >::~unique_ptr[abi:v15000]() (this=<optimized out>) at ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259
2024-12-17 01:15:58 No locals.
2024-12-17 01:15:58 #17 DB::CachedOnDiskReadBufferFromFile::~CachedOnDiskReadBufferFromFile (this=0x618002d62498) at ./src/Disks/IO/CachedOnDiskReadBufferFromFile.cpp:535
2024-12-17 01:15:58 No locals.
2024-12-17 01:15:58 #18 0x000055d8ac4d13b9 in std::__1::__shared_count::__release_shared[abi:v15000]() (this=0x618002d62480) at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174
2024-12-17 01:15:58 No locals.
2024-12-17 01:15:58 #19 std::__1::__shared_weak_count::__release_shared[abi:v15000]() (this=0x618002d62480) at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215
2024-12-17 01:15:58 No locals.
2024-12-17 01:15:58 #20 std::__1::shared_ptr<DB::SeekableReadBuffer>::~shared_ptr[abi:v15000]() (this=0x616001f955c8) at ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702
| |
| Test script exit code | OK |
| Server successfully started | OK |
| No sanitizer asserts | OK |
| No OOM messages in clickhouse-server.log | OK |
| No logical errors | OK |
| No lost s3 keys | OK |
| Not crashed | OK |
| No OOM in dmesg | OK |
