🌚🌞

ClickHouse Stress test (debug) for PR #2011

check_status.tsv Commit clickhouse-server.err.log clickhouse-server.final.log clickhouse-server.initial.log clickhouse-server.log clickhouse-server.stress.log coordination.tar dmesg.log fatal_messages.txt gdb.log metric_log.tsv.zst query_log.tsv.zst run.log stderr.log stdout.log stress_run_logs.tar.zst test_results.tsv trace_log.tsv.zst Task (github actions) Job (github actions)

Test nameTest status
Possible deadlock on shutdown (see gdb.log) FAIL
Killed by signal (in clickhouse-server.log) FAIL 
 [ 21080 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: B60A50EC378B255BCB1BCE3E8293DB3AF7C1EF26, git hash: 6a78805ec9737295beb4a8d213873fb4093d11eb) (from thread 21591) Terminate called for uncaught exception:
 [ 21080 ] {} <Fatal> BaseDaemon: Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):
 [ 21080 ] {} <Fatal> BaseDaemon: 
 [ 21080 ] {} <Fatal> BaseDaemon: 0. /build/contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000164a1a32
 [ 21080 ] {} <Fatal> BaseDaemon: 1. /build/src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c653179
 [ 21080 ] {} <Fatal> BaseDaemon: 2. /build/contrib/llvm-project/libcxx/include/string:1499: _ZN12Coordination9ExceptionC2IRA16_KcQsr3stdE16is_convertible_vIT_NSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEEEOS5_NS_5ErrorE @ 0x00000000134a55fb
 [ 21080 ] {} <Fatal> BaseDaemon: 3. /build/src/Common/ZooKeeper/IKeeper.h:0: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000134ccdc2
 [ 21080 ] {} <Fatal> BaseDaemon: 4. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1365: Coordination::ZooKeeper::exists(String const&, std::function<void (Coordination::ExistsResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x00000000134cf245
 [ 21080 ] {} <Fatal> BaseDaemon: 5. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001347e7b8
 [ 21080 ] {} <Fatal> BaseDaemon: 6. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001347e3b9
 [ 21080 ] {} <Fatal> BaseDaemon: 7. /build/src/Common/ZooKeeper/ZooKeeper.cpp:598: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000001347e973
 [ 21080 ] {} <Fatal> BaseDaemon: 8. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001347c488
 [ 21080 ] {} <Fatal> BaseDaemon: 9. /build/src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::forcefullyRemoveBrokenOutdatedPartFromZooKeeperBeforeDetaching(String const&) @ 0x000000001241f508
 [ 21080 ] {} <Fatal> BaseDaemon: 10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::MergeTreeData::loadOutdatedDataParts(bool)::$_0, void ()>>(std::__function::__policy_stor
 [ 21080 ] {} <Fatal> BaseDaemon: 11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x000000001041557a
 [ 21080 ] {} <Fatal> BaseDaemon: 12. /build/contrib/llvm-project/libcxx/include/future:2069: std::packaged_task<void ()>::operator()() @ 0x000000000cbaa497
 [ 21080 ] {} <Fatal> BaseDaemon: 13. /build/src/Common/ThreadPool.cpp:781: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000c709f9f
 [ 21080 ] {} <Fatal> BaseDaemon: 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x000000000c70f122
 [ 21080 ] {} <Fatal> BaseDaemon: 15. /build/base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000c707d2e
 [ 21080 ] {} <Fatal> BaseDaemon: 16. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)()
 [ 21080 ] {} <Fatal> BaseDaemon: 17. ? @ 0x00007f89f86b2ac3
 [ 21080 ] {} <Fatal> BaseDaemon: 18. ? @ 0x00007f89f87448d0
 [ 21080 ] {} <Fatal> BaseDaemon:  (version 24.8.14.10546.altinitytest (altinity build))
 [ 21961 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
 [ 21961 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: B60A50EC378B255BCB1BCE3E8293DB3AF7C1EF26, git hash: 6a78805ec9737295beb4a8d213873fb4093d11eb) (from thread 21591) Received signal 6
 [ 21961 ] {} <Fatal> BaseDaemon: Signal description: Aborted
 [ 21961 ] {} <Fatal> BaseDaemon: 
 [ 21961 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000c681388 0x000000000c8d475e 0x00007f89f8660520 0x00007f89f86b49fd 0x00007f89f8660476 0x00007f89f86467f3 0x000000000c8d4d2e 0x000000001855f883 0x000000001855f818 0x00000000127abe0a 0x000000001097a034 0x000000001097cc94 0x000000001097d267 0x0000
 [ 21961 ] {} <Fatal> BaseDaemon: ########################################
 [ 21961 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: B60A50EC378B255BCB1BCE3E8293DB3AF7C1EF26, git hash: 6a78805ec9737295beb4a8d213873fb4093d11eb) (from thread 21591) (no query) Received signal Aborted (6)
 [ 21961 ] {} <Fatal> BaseDaemon: 
 [ 21961 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000c681388 0x000000000c8d475e 0x00007f89f8660520 0x00007f89f86b49fd 0x00007f89f8660476 0x00007f89f86467f3 0x000000000c8d4d2e 0x000000001855f883 0x000000001855f818 0x00000000127abe0a 0x000000001097a034 0x000000001097cc94 0x000000001097d267 0x0000
 [ 21961 ] {} <Fatal> BaseDaemon: 0.0. inlined from /build/src/Common/StackTrace.cpp:349: StackTrace::tryCapture()
 [ 21961 ] {} <Fatal> BaseDaemon: 0. /build/src/Common/StackTrace.cpp:318: StackTrace::StackTrace(ucontext_t const&) @ 0x000000000c681388
 [ 21961 ] {} <Fatal> BaseDaemon: 1. /build/src/Common/SignalHandlers.cpp:0: signalHandler(int, siginfo_t*, void*) @ 0x000000000c8d475e
 [ 21961 ] {} <Fatal> BaseDaemon: 2. ? @ 0x00007f89f8660520
 [ 21961 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00007f89f86b49fd
 [ 21961 ] {} <Fatal> BaseDaemon: 4. ? @ 0x00007f89f8660476
 [ 21961 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00007f89f86467f3
 [ 21961 ] {} <Fatal> BaseDaemon: 6. /build/src/Common/SignalHandlers.cpp:140: terminate_handler() @ 0x000000000c8d4d2e
 [ 21961 ] {} <Fatal> BaseDaemon: 7. /build/contrib/llvm-project/libcxxabi/src/cxa_handlers.cpp:61: std::__terminate(void (*)()) @ 0x000000001855f883
 [ 21961 ] {} <Fatal> BaseDaemon: 8. /build/contrib/llvm-project/libcxxabi/src/cxa_handlers.cpp:79: std::terminate() @ 0x000000001855f818
 [ 21961 ] {} <Fatal> BaseDaemon: 9. /build/src/Storages/MergeTree/MergeTreeData.cpp:0: DB::MergeTreeData::loadOutdatedDataParts(bool) @ 0x00000000127abe0a
 [ 21961 ] {} <Fatal> BaseDaemon: 10.0. inlined from /build/src/Common/Stopwatch.h:86: Stopwatch::nanoseconds() const
 [ 21961 ] {} <Fatal> BaseDaemon: 10.1. inlined from /build/src/Common/Stopwatch.h:72: Stopwatch::elapsedNanoseconds() const
 [ 21961 ] {} <Fatal> BaseDaemon: 10.2. inlined from /build/src/Common/Stopwatch.h:74: Stopwatch::elapsedMilliseconds() const
 [ 21961 ] {} <Fatal> BaseDaemon: 10. /build/src/Core/BackgroundSchedulePool.cpp:113: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x000000001097a034
 [ 21961 ] {} <Fatal> BaseDaemon: 11.0. inlined from /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: ~shared_ptr
 [ 21961 ] {} <Fatal> BaseDaemon: 11. /build/src/Core/BackgroundSchedulePool.cpp:305: DB::BackgroundSchedulePool::threadFunction() @ 0x000000001097cc94
 [ 21961 ] {} <Fatal> BaseDaemon: 12.0. inlined from /build/src/Core/BackgroundSchedulePool.cpp:0: operator()
 [ 21961 ] {} <Fatal> BaseDaemon: 12.1. inlined from /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ?
 [ 21961 ] {} <Fatal> BaseDaemon: 12.2. inlined from /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:479: ?
 [ 21961 ] {} <Fatal> BaseDaemon: 12.3. inlined from /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: ?
 [ 21961 ] {} <Fatal> BaseDaemon: 12. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001097d267
 [ 21961 ] {} <Fatal> BaseDaemon: 13.0. inlined from /build/base/base/../base/wide_integer_impl.h:817: bool wide::integer<128ul, unsigned int>::_impl::operator_eq<wide::integer<128ul, unsigned int>>(wide::integer<128ul, unsigned int> const&, wide::integer<128ul, unsigned int> const&)
 [ 21961 ] {} <Fatal> BaseDaemon: 13.1. inlined from /build/base/base/../base/wide_integer_impl.h:1491: bool wide::operator==<128ul, unsigned int, 128ul, unsigned int>(wide::integer<128ul, unsigned int> const&, wide::integer<128ul, unsigned int> const&)
 [ 21961 ] {} <Fatal> BaseDaemon: 13.2. inlined from /build/base/base/../base/strong_typedef.h:42: StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>::operator==(StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag> const&) const
 [ 21961 ] {} <Fatal> BaseDaemon: 13.3. inlined from /build/src/Common/OpenTelemetryTraceContext.h:65: DB::OpenTelemetry::Span::isTraceEnabled() const
 [ 21961 ] {} <Fatal> BaseDaemon: 13. /build/src/Common/ThreadPool.cpp:789: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000c707d2e
 [ 21961 ] {} <Fatal> BaseDaemon: 14.0. inlined from /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), Threa
 [ 21961 ] {} <Fatal> BaseDaemon: 14.1. inlined from /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259: ~unique_ptr
 [ 21961 ] {} <Fatal> BaseDaemon: 14. /build/contrib/llvm-project/libcxx/include/thread:297: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImp
 [ 21961 ] {} <Fatal> BaseDaemon: 15. ? @ 0x00007f89f86b2ac3
 [ 21961 ] {} <Fatal> BaseDaemon: 16. ? @ 0x00007f89f87448d0
 [ 21961 ] {} <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: 9E649BC96D94D019317B489C2F38D15D)
 [ 21961 ] {} <Fatal> BaseDaemon: ClickHouse version 24.8.14.10546.altinitytest is old and should be upgraded to the latest version.
 [ 21074 ] {} <Fatal> Application: Child process was terminated by signal 6.
 [ 21080 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: B60A50EC378B255BCB1BCE3E8293DB3AF7C1EF26, git hash: 6a78805ec9737295beb4a8d213873fb4093d11eb) (from thread 21591) Terminate called for uncaught exception:
 [ 21080 ] {} <Fatal> BaseDaemon: Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):
 [ 21080 ] {} <Fatal> BaseDaemon: 
 [ 21080 ] {} <Fatal> BaseDaemon: 0. /build/contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x00000000164a1a32
 [ 21080 ] {} <Fatal> BaseDaemon: 1. /build/src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c653179
 [ 21080 ] {} <Fatal> BaseDaemon: 2. /build/contrib/llvm-project/libcxx/include/string:1499: _ZN12Coordination9ExceptionC2IRA16_KcQsr3stdE16is_convertible_vIT_NSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEEEEEOS5_NS_5ErrorE @ 0x00000000134a55fb
 [ 21080 ] {} <Fatal> BaseDaemon: 3. /build/src/Common/ZooKeeper/IKeeper.h:0: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000134ccdc2
 [ 21080 ] {} <Fatal> BaseDaemon: 4. /build/src/Common/ZooKeeper/ZooKeeperImpl.cpp:1365: Coordination::ZooKeeper::exists(String const&, std::function<void (Coordination::ExistsResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x00000000134cf245
 [ 21080 ] {} <Fatal> BaseDaemon: 5. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001347e7b8
 [ 21080 ] {} <Fatal> BaseDaemon: 6. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001347e3b9
 [ 21080 ] {} <Fatal> BaseDaemon: 7. /build/src/Common/ZooKeeper/ZooKeeper.cpp:598: zkutil::ZooKeeper::existsWatch(String const&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x000000001347e973
 [ 21080 ] {} <Fatal> BaseDaemon: 8. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001347c488
 [ 21080 ] {} <Fatal> BaseDaemon: 9. /build/src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::forcefullyRemoveBrokenOutdatedPartFromZooKeeperBeforeDetaching(String const&) @ 0x000000001241f508
 [ 21080 ] {} <Fatal> BaseDaemon: 10. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::MergeTreeData::loadOutdatedDataParts(bool)::$_0, void ()>>(std::__function::__policy_stor
 [ 21080 ] {} <Fatal> BaseDaemon: 11. /build/contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x000000001041557a
 [ 21080 ] {} <Fatal> BaseDaemon: 12. /build/contrib/llvm-project/libcxx/include/future:2069: std::packaged_task<void ()>::operator()() @ 0x000000000cbaa497
 [ 21080 ] {} <Fatal> BaseDaemon: 13. /build/src/Common/ThreadPool.cpp:781: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000c709f9f
 [ 21080 ] {} <Fatal> BaseDaemon: 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:0: ? @ 0x000000000c70f122
 [ 21080 ] {} <Fatal> BaseDaemon: 15. /build/base/base/../base/wide_integer_impl.h:817: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000c707d2e
 [ 21080 ] {} <Fatal> BaseDaemon: 16. /build/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)()
 [ 21080 ] {} <Fatal> BaseDaemon: 17. ? @ 0x00007f89f86b2ac3
 [ 21080 ] {} <Fatal> BaseDaemon: 18. ? @ 0x00007f89f87448d0
 [ 21080 ] {} <Fatal> BaseDaemon:  (version 24.8.14.10546.altinitytest (altinity build))
 [ 21961 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
 [ 21961 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: B60A50EC378B255BCB1BCE3E8293DB3AF7C1EF26, git hash: 6a78805ec9737295beb4a8d213873fb4093d11eb) (from thread 21591) Received signal 6
 [ 21961 ] {} <Fatal> BaseDaemon: Signal description: Aborted
 [ 21961 ] {} <Fatal> BaseDaemon: 
 [ 21961 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000c681388 0x000000000c8d475e 0x00007f89f8660520 0x00007f89f86b49fd 0x00007f89f8660476 0x00007f89f86467f3 0x000000000c8d4d2e 0x000000001855f883 0x000000001855f818 0x00000000127abe0a 0x000000001097a034 0x000000001097cc94 0x000000001097d267 0x0000
 [ 21961 ] {} <Fatal> BaseDaemon: ########################################
 [ 21961 ] {} <Fatal> BaseDaemon: (version 24.8.14.10546.altinitytest (altinity build), build id: B60A50EC378B255BCB1BCE3E8293DB3AF7C1EF26, git hash: 6a78805ec9737295beb4a8d213873fb4093d11eb) (from thread 21591) (no query) Received signal Aborted (6)
 [ 21961 ] {} <Fatal> BaseDaemon: 
 [ 21961 ] {} <Fatal> BaseDaemon: Stack trace: 0x000000000c681388 0x000000000c8d475e 0x00007f89f8660520 0x00007f89f86b49fd 0x00007f89f8660476 0x00007f89f86467f3 0x000000000c8d4d2e 0x000000001855f883 0x000000001855f818 0x00000000127abe0a 0x000000001097a034 0x000000001097cc94 0x000000001097d267 0x0000
 [ 21961 ] {} <Fatal> BaseDaemon: 0.0. inlined from /build/src/Common/StackTrace.cpp:349: StackTrace::tryCapture()
Killed by signal (output files) FAIL
Found signal in gdb.log FAIL
No queries hung OK
Test script exit code OK
Server successfully started OK
Warning: server did not stop yet OK
No sanitizer asserts OK
No OOM messages in clickhouse-server.log OK
No logical errors OK
No lost s3 keys OK
No SharedMergeTree lost forever in clickhouse-server.log OK
No OOM in dmesg OK