2024.09.24 01:07:23.057531 [ 339 ] {} Context: Server was built in debug mode. It will work slowly. 2024.09.24 01:07:23.059953 [ 339 ] {} Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. 2024.09.24 01:07:26.779663 [ 339 ] {} Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182 2024.09.24 01:07:27.500040 [ 339 ] {} KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper. 2024.09.24 01:07:27.500738 [ 339 ] {} KeeperLogStore: Removing all changelogs 2024.09.24 01:07:27.502034 [ 339 ] {} KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647. 2024.09.24 01:07:27.506538 [ 339 ] {} KeeperStateManager: No state was read 2024.09.24 01:07:27.506628 [ 339 ] {} RaftInstance: invalid election timeout lower bound detected, adjusted to 0 2024.09.24 01:07:27.506705 [ 339 ] {} RaftInstance: invalid election timeout upper bound detected, adjusted to 0 2024.09.24 01:07:27.558032 [ 597 ] {} RaftInstance: Election timeout, initiate leader election 2024.09.24 01:07:27.636815 [ 339 ] {} Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist 2024.09.24 01:07:27.636926 [ 339 ] {} Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/ 2024.09.24 01:07:27.984260 [ 339 ] {} Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring. 2024.09.24 01:08:08.577873 [ 600 ] {} RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled 2024.09.24 01:08:08.590152 [ 595 ] {} KeeperLogStore: Raft server is not set in LogStore. 2024.09.24 01:08:08.660699 [ 339 ] {} KeeperLogStore: Changelog is shut down 2024.09.24 01:08:10.884500 [ 775 ] {} Context: Server was built in debug mode. It will work slowly. 2024.09.24 01:08:10.886870 [ 775 ] {} Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. 2024.09.24 01:08:14.627052 [ 775 ] {} Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182 2024.09.24 01:08:15.346192 [ 775 ] {} KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647. 2024.09.24 01:08:15.350084 [ 775 ] {} RaftInstance: invalid election timeout lower bound detected, adjusted to 0 2024.09.24 01:08:15.350170 [ 775 ] {} RaftInstance: invalid election timeout upper bound detected, adjusted to 0 2024.09.24 01:08:15.401498 [ 995 ] {} RaftInstance: Election timeout, initiate leader election 2024.09.24 01:08:16.516170 [ 775 ] {} Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring. 2024.09.24 01:12:53.484025 [ 777 ] {08d8c38e-3db0-4f9e-8fe2-41409a92b541} executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:39624) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 = (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8a1d99 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee95fa3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001eea48e6 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee51b4c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee780df in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbcd8cb in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8fb96 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f081 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc8dec5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc9173d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f0b7 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 31. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 2024.09.24 01:12:53.484933 [ 777 ] {08d8c38e-3db0-4f9e-8fe2-41409a92b541} TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8a1d99 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee95fa3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001eea48e6 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee51b4c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee780df in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbcd8cb in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8fb96 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f081 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc8dec5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc9173d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f0b7 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 31. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 2024.09.24 01:12:53.644589 [ 777 ] {1e7aae22-90e5-4c40-bf9d-204a7f32bf53} executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:39624) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8a1d99 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee95fa3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001eea48e6 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee51b4c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee780df in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbcd8cb in /usr/bin/clickhouse 13. /build/src/Interpreters/executeQuery.cpp:1085: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001d0b0e97 in /usr/bin/clickhouse 14. /build/src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x000000001d0ac1a4 in /usr/bin/clickhouse 15. /build/src/Server/TCPHandler.cpp:496: DB::TCPHandler::runImpl() @ 0x000000001e64374d in /usr/bin/clickhouse 16. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 22. ? @ 0x00007fb34c116ac3 in ? 23. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:12:53.645444 [ 777 ] {1e7aae22-90e5-4c40-bf9d-204a7f32bf53} TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8a1d99 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee95fa3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001eea48e6 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee51b4c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee780df in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbcd8cb in /usr/bin/clickhouse 13. /build/src/Interpreters/executeQuery.cpp:1085: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001d0b0e97 in /usr/bin/clickhouse 14. /build/src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x000000001d0ac1a4 in /usr/bin/clickhouse 15. /build/src/Server/TCPHandler.cpp:496: DB::TCPHandler::runImpl() @ 0x000000001e64374d in /usr/bin/clickhouse 16. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 22. ? @ 0x00007fb34c116ac3 in ? 23. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:12:53.904297 [ 777 ] {f35e4b9a-ce6d-40ec-a359-0ecfb04e1eb1} executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:39624) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) FROM test.hits WHERE NOT ignore(AdvEngineID);), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:12:53.911659 [ 777 ] {f35e4b9a-ce6d-40ec-a359-0ecfb04e1eb1} TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:12:54.105803 [ 777 ] {aeedae65-1e89-4082-b030-8ab8cb2ac0ca} executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:39624) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID > (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8a1d99 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee95fa3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001eea48e6 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee51b4c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee780df in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbcd8cb in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8fb96 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f081 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc8dec5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc9173d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f0b7 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc8dec5 in /usr/bin/clickhouse 31. /build/src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr&, std::shared_ptr, unsigned long, std::map, std::allocator>>&, std::map, std::allocator>>&, bool, bool) @ 0x000000001cfa05e8 in /usr/bin/clickhouse 2024.09.24 01:12:54.106771 [ 777 ] {aeedae65-1e89-4082-b030-8ab8cb2ac0ca} TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8a1d99 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee95fa3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001eea48e6 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee51b4c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee780df in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbcd8cb in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8fb96 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f081 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc8dec5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc9173d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8f0b7 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc8e1e0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3c1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc8e466 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc8e3ce in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc8e16a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc8dec5 in /usr/bin/clickhouse 31. /build/src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr&, std::shared_ptr, unsigned long, std::map, std::allocator>>&, std::map, std::allocator>>&, bool, bool) @ 0x000000001cfa05e8 in /usr/bin/clickhouse 2024.09.24 01:12:54.266462 [ 777 ] {3937bc8e-9a34-4253-bf16-561e66007ab0} executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:39624) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID < (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:12:54.271826 [ 777 ] {3937bc8e-9a34-4253-bf16-561e66007ab0} TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b8a1f28 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1a4c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:21:28.492510 [ 1889 ] {789b3801-25c9-4f7c-9103-22b1b87c1cec} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.187531016 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45658) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:28.492275 [ 1915 ] {39661e5a-eed1-45a9-b193-56f931de6047} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.951998831 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47976) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:29.468113 [ 2661 ] {c08adf16-ed9f-4a6b-a4a6-b6c40617e7b3} executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 579.51 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:60574) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b8a2148 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1c27 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:21:29.575894 [ 1915 ] {39661e5a-eed1-45a9-b193-56f931de6047} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:47976). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:29.575943 [ 1889 ] {789b3801-25c9-4f7c-9103-22b1b87c1cec} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.187531016 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:29.576037 [ 2661 ] {c08adf16-ed9f-4a6b-a4a6-b6c40617e7b3} TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 579.51 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b8a2148 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1c27 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:21:29.576353 [ 1915 ] {39661e5a-eed1-45a9-b193-56f931de6047} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.951998831 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:29.577077 [ 1915 ] {39661e5a-eed1-45a9-b193-56f931de6047} TCPHandler: Client has gone away. 2024.09.24 01:21:33.361525 [ 1889 ] {2c947310-fdaf-473e-a42d-4ece72fd56ea} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.099994578 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:56748) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:33.428233 [ 1915 ] {74e9a1d4-80bb-4f5f-a61a-d37e757366fa} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.524270756 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45060) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:33.434875 [ 2663 ] {635ce5a1-1fc0-4bd4-906c-036c12a4f679} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.524806117 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44742) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:33.438909 [ 2661 ] {fb289f5f-b862-4ccc-9cf8-113ad126ea3c} executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 588.65 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:60574) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b8a2148 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1c27 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:21:33.488688 [ 1915 ] {74e9a1d4-80bb-4f5f-a61a-d37e757366fa} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:45060). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:33.489142 [ 1915 ] {74e9a1d4-80bb-4f5f-a61a-d37e757366fa} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.524270756 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:33.489729 [ 1915 ] {74e9a1d4-80bb-4f5f-a61a-d37e757366fa} TCPHandler: Client has gone away. 2024.09.24 01:21:33.510362 [ 2661 ] {fb289f5f-b862-4ccc-9cf8-113ad126ea3c} TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 588.65 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b8a2148 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1c27 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:21:34.424675 [ 1889 ] {2c947310-fdaf-473e-a42d-4ece72fd56ea} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.099994578 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:34.957125 [ 2663 ] {635ce5a1-1fc0-4bd4-906c-036c12a4f679} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:44742). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:34.957616 [ 2663 ] {635ce5a1-1fc0-4bd4-906c-036c12a4f679} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.524806117 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:34.993391 [ 2663 ] {635ce5a1-1fc0-4bd4-906c-036c12a4f679} TCPHandler: Client has gone away. 2024.09.24 01:21:36.207276 [ 2661 ] {359d2a7f-4451-45dd-913d-9c9068235950} executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 577.49 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:60574) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits PREWHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b8a2148 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1c27 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:21:36.285192 [ 2661 ] {359d2a7f-4451-45dd-913d-9c9068235950} TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 577.49 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b8a2148 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b8a1c27 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b878204 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 2024.09.24 01:21:37.809805 [ 1889 ] {439f7f24-087f-44cc-ae07-dbc730b58c77} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.553674198 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:56760) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:37.814708 [ 1915 ] {ec7b2364-2fbf-4f81-bfb1-61fdbbb83c9f} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.340667772 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45066) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:37.820926 [ 2663 ] {428c8b97-4b23-4887-b9ce-6580aa016473} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.346091848 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44758) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:37.820977 [ 2661 ] {0ff29777-ffdb-4ccf-9b4d-7fbba7bd729f} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.341226745 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43014) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:38.671343 [ 1915 ] {ec7b2364-2fbf-4f81-bfb1-61fdbbb83c9f} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.340667772 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:38.677560 [ 1915 ] {ec7b2364-2fbf-4f81-bfb1-61fdbbb83c9f} TCPHandler: Client has gone away. 2024.09.24 01:21:38.688155 [ 2661 ] {0ff29777-ffdb-4ccf-9b4d-7fbba7bd729f} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:43014). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:38.688718 [ 2661 ] {0ff29777-ffdb-4ccf-9b4d-7fbba7bd729f} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.341226745 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:38.700300 [ 2661 ] {0ff29777-ffdb-4ccf-9b4d-7fbba7bd729f} TCPHandler: Client has gone away. 2024.09.24 01:21:38.908254 [ 1889 ] {439f7f24-087f-44cc-ae07-dbc730b58c77} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.553674198 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:38.993539 [ 2663 ] {428c8b97-4b23-4887-b9ce-6580aa016473} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.346091848 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:38.996492 [ 2663 ] {428c8b97-4b23-4887-b9ce-6580aa016473} TCPHandler: Client has gone away. 2024.09.24 01:21:42.598211 [ 2661 ] {8bece04a-388d-47da-bde9-b5fbfe2a7ab0} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348491085 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47222) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:42.684961 [ 1889 ] {46e583a0-ee4b-44ce-90b1-f11bc9c83e95} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.067994737 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:47222) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:42.710200 [ 2661 ] {8bece04a-388d-47da-bde9-b5fbfe2a7ab0} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348491085 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:42.711084 [ 2661 ] {8bece04a-388d-47da-bde9-b5fbfe2a7ab0} TCPHandler: Client has gone away. 2024.09.24 01:21:42.711675 [ 1915 ] {1681ea30-0f07-46d7-b689-642e7396c8d7} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.039994875 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39856) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:42.714035 [ 2663 ] {2a6bd59c-8421-4bcf-b77c-f9c9cbc47a87} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.493812792 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50872) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:42.715643 [ 3396 ] {3351eccb-e658-4ad6-b9f1-c509e39c1ae3} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.487395218 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36084) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:42.892447 [ 2663 ] {2a6bd59c-8421-4bcf-b77c-f9c9cbc47a87} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:50872). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:42.892886 [ 2663 ] {2a6bd59c-8421-4bcf-b77c-f9c9cbc47a87} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.493812792 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:42.896168 [ 2663 ] {2a6bd59c-8421-4bcf-b77c-f9c9cbc47a87} TCPHandler: Client has gone away. 2024.09.24 01:21:43.167315 [ 3396 ] {3351eccb-e658-4ad6-b9f1-c509e39c1ae3} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:36084). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:43.167699 [ 3396 ] {3351eccb-e658-4ad6-b9f1-c509e39c1ae3} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.487395218 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:43.168303 [ 3396 ] {3351eccb-e658-4ad6-b9f1-c509e39c1ae3} TCPHandler: Client has gone away. 2024.09.24 01:21:43.216060 [ 1915 ] {1681ea30-0f07-46d7-b689-642e7396c8d7} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:39856). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:43.216530 [ 1915 ] {1681ea30-0f07-46d7-b689-642e7396c8d7} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.039994875 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:43.217377 [ 1915 ] {1681ea30-0f07-46d7-b689-642e7396c8d7} TCPHandler: Client has gone away. 2024.09.24 01:21:43.221246 [ 1889 ] {46e583a0-ee4b-44ce-90b1-f11bc9c83e95} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.067994737 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:46.755391 [ 3396 ] {c36d00a8-fe64-49a7-98e5-df5aa80d5a59} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.438997165 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50874) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.756784 [ 2661 ] {9aafe127-4cb9-48c4-98a3-4d0221ac0926} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.491270267 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39862) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.757157 [ 2663 ] {7ef43b4e-5218-47c1-8b06-801b63c164f9} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.429218844 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60452) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.756889 [ 1889 ] {0fdf722a-4cdd-45f5-84ca-67b99f0873dd} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.87267566 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:47226) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.762643 [ 3400 ] {24d88db6-8d35-4722-8d5f-d16a39be3605} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.509203483 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36100) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.779026 [ 2661 ] {9aafe127-4cb9-48c4-98a3-4d0221ac0926} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.491270267 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.784576 [ 2663 ] {7ef43b4e-5218-47c1-8b06-801b63c164f9} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.429218844 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.804331 [ 1915 ] {08e5bd5f-73f1-48e3-82d7-05370acfbc61} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.591636881 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47226) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.889911 [ 2663 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:60452). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013cee00c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ced280 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b2189bc in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e6425a1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 15. ? @ 0x00007fb34c116ac3 in ? 16. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:46.892334 [ 2661 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:39862). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013cee00c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ced280 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b2189bc in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e6425a1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 15. ? @ 0x00007fb34c116ac3 in ? 16. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:46.928177 [ 3396 ] {c36d00a8-fe64-49a7-98e5-df5aa80d5a59} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.438997165 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:46.973827 [ 3396 ] {c36d00a8-fe64-49a7-98e5-df5aa80d5a59} TCPHandler: Client has gone away. 2024.09.24 01:21:48.048483 [ 3400 ] {24d88db6-8d35-4722-8d5f-d16a39be3605} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:36100). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:48.049214 [ 3400 ] {24d88db6-8d35-4722-8d5f-d16a39be3605} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.509203483 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:48.054476 [ 3400 ] {24d88db6-8d35-4722-8d5f-d16a39be3605} TCPHandler: Client has gone away. 2024.09.24 01:21:48.152346 [ 1915 ] {08e5bd5f-73f1-48e3-82d7-05370acfbc61} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.591636881 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:48.153991 [ 1915 ] {08e5bd5f-73f1-48e3-82d7-05370acfbc61} TCPHandler: Client has gone away. 2024.09.24 01:21:48.185341 [ 1889 ] {0fdf722a-4cdd-45f5-84ca-67b99f0873dd} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.87267566 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:51.722181 [ 3400 ] {06d4144a-e3c4-44e6-b1a0-2e1b139e6783} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.030454279 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50912) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:51.769140 [ 3400 ] {06d4144a-e3c4-44e6-b1a0-2e1b139e6783} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.030454279 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:51.964161 [ 1915 ] {502d0fd8-c6e7-4191-b166-4844622fbe08} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.485188681 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:40638) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.039044 [ 1889 ] {6c4e414e-d5a4-44a4-89a3-973f8ba65663} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.013928596 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:37122) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.151115 [ 2661 ] {a30c77bd-4237-4c7a-8b60-9234ac87d5c3} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.698611588 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36962) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.151251 [ 3400 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:50912). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013cee00c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ced280 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b2189bc in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e6425a1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 15. ? @ 0x00007fb34c116ac3 in ? 16. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:52.170336 [ 3402 ] {b73a840e-b7ad-432a-b660-5c277eac18f8} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.64956917 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35130) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.240543 [ 3396 ] {fcc2d636-fafd-4a31-a43b-a912224aacd1} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.542271669 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:56956) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.246314 [ 2663 ] {87b1aa4a-a6e2-45f4-9690-42e9357abcb8} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.043994855 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48710) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:52.246490 [ 1915 ] {502d0fd8-c6e7-4191-b166-4844622fbe08} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:40638). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:52.246997 [ 1915 ] {502d0fd8-c6e7-4191-b166-4844622fbe08} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.485188681 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.275092 [ 1915 ] {502d0fd8-c6e7-4191-b166-4844622fbe08} TCPHandler: Client has gone away. 2024.09.24 01:21:52.362535 [ 2661 ] {a30c77bd-4237-4c7a-8b60-9234ac87d5c3} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.698611588 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.365716 [ 2661 ] {a30c77bd-4237-4c7a-8b60-9234ac87d5c3} TCPHandler: Client has gone away. 2024.09.24 01:21:52.388274 [ 3402 ] {b73a840e-b7ad-432a-b660-5c277eac18f8} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.64956917 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:52.401495 [ 3402 ] {b73a840e-b7ad-432a-b660-5c277eac18f8} TCPHandler: Client has gone away. 2024.09.24 01:21:53.264411 [ 2663 ] {87b1aa4a-a6e2-45f4-9690-42e9357abcb8} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.043994855 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:53.274113 [ 2663 ] {87b1aa4a-a6e2-45f4-9690-42e9357abcb8} TCPHandler: Client has gone away. 2024.09.24 01:21:53.286021 [ 1889 ] {6c4e414e-d5a4-44a4-89a3-973f8ba65663} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.013928596 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:53.359124 [ 3396 ] {fcc2d636-fafd-4a31-a43b-a912224aacd1} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.542271669 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:21:53.361021 [ 3396 ] {fcc2d636-fafd-4a31-a43b-a912224aacd1} TCPHandler: Client has gone away. 2024.09.24 01:21:57.734091 [ 3400 ] {77231bc5-5215-4116-9e94-a05171155a8a} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.067994737 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50928) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.017078 [ 3400 ] {77231bc5-5215-4116-9e94-a05171155a8a} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.067994737 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.040617 [ 2663 ] {9a9ed47f-7068-4f32-9cc4-b85c2a0130de} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.275993712 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:56970) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.063864 [ 3402 ] {756affd6-a165-4ca7-b436-ad315db51655} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.183994164 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48718) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.063889 [ 1889 ] {d1acd035-f411-4dbe-9cb7-3b6dd71949de} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.527992469 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:37138) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.063999 [ 3396 ] {5cfc5bdb-208e-4916-b0f7-2c59a9c20d3b} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.559992312 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:40642) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.070076 [ 1915 ] {68f1354b-3b65-4240-8825-a01e93bef884} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.231993928 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36968) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.091902 [ 3404 ] {495f8cd6-73c3-4042-9ca0-795bbf6631c1} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.415993022 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35144) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.092980 [ 2661 ] {e08608c3-5c68-4846-8b4e-842e53a0676f} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.455992824 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47826) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.175509 [ 3400 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:50928). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013cee00c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ced280 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b2189bc in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e6425a1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 15. ? @ 0x00007fb34c116ac3 in ? 16. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:58.410183 [ 2663 ] {9a9ed47f-7068-4f32-9cc4-b85c2a0130de} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:56970). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:58.410690 [ 2663 ] {9a9ed47f-7068-4f32-9cc4-b85c2a0130de} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.275993712 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.418080 [ 1915 ] {68f1354b-3b65-4240-8825-a01e93bef884} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.231993928 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.437902 [ 1915 ] {68f1354b-3b65-4240-8825-a01e93bef884} TCPHandler: Client has gone away. 2024.09.24 01:21:58.443768 [ 2663 ] {9a9ed47f-7068-4f32-9cc4-b85c2a0130de} TCPHandler: Client has gone away. 2024.09.24 01:21:58.589900 [ 3402 ] {756affd6-a165-4ca7-b436-ad315db51655} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.183994164 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:58.602020 [ 3402 ] {756affd6-a165-4ca7-b436-ad315db51655} TCPHandler: Client has gone away. 2024.09.24 01:21:59.009242 [ 3404 ] {495f8cd6-73c3-4042-9ca0-795bbf6631c1} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:35144). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:59.009949 [ 3404 ] {495f8cd6-73c3-4042-9ca0-795bbf6631c1} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.415993022 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:59.036824 [ 3404 ] {495f8cd6-73c3-4042-9ca0-795bbf6631c1} TCPHandler: Client has gone away. 2024.09.24 01:21:59.084248 [ 3396 ] {5cfc5bdb-208e-4916-b0f7-2c59a9c20d3b} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:40642). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013cefb88 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ceef04 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b204acd in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e655c35 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e64c2df in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e6448d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 17. ? @ 0x00007fb34c116ac3 in ? 18. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:21:59.084751 [ 3396 ] {5cfc5bdb-208e-4916-b0f7-2c59a9c20d3b} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.559992312 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:59.085994 [ 3396 ] {5cfc5bdb-208e-4916-b0f7-2c59a9c20d3b} TCPHandler: Client has gone away. 2024.09.24 01:21:59.751619 [ 2661 ] {e08608c3-5c68-4846-8b4e-842e53a0676f} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.455992824 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:21:59.789073 [ 2661 ] {e08608c3-5c68-4846-8b4e-842e53a0676f} TCPHandler: Client has gone away. 2024.09.24 01:21:59.909592 [ 1889 ] {d1acd035-f411-4dbe-9cb7-3b6dd71949de} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.527992469 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:03.182383 [ 3396 ] {714b163a-8080-4c7e-8444-38518bcb1d9a} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.153168677 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50000) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:03.211282 [ 3396 ] {714b163a-8080-4c7e-8444-38518bcb1d9a} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.153168677 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:04.015995 [ 2661 ] {cb3174ee-2252-4697-9ab0-87e8400880b9} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.235217688 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45904) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:04.111212 [ 3404 ] {f4c959eb-f4f9-426b-81bb-dd00d86e68d1} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.53599243 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53140) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.124080 [ 3396 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:50000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013cee00c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ced280 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b2189bc in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e6425a1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 15. ? @ 0x00007fb34c116ac3 in ? 16. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:22:04.126781 [ 2663 ] {6a66e478-ce81-47b1-8428-dd7a538db26f} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.179994184 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43458) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.150423 [ 1889 ] {c4b3e206-7fe4-4108-b558-d8537693544a} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.275993711 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35938) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.155909 [ 1915 ] {ed9c9d9e-dfd3-4152-8002-6274c24d9cb5} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.331993436 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33346) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.157413 [ 3402 ] {7f58ff5c-ac29-4978-bdac-ee1d0eca2866} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.023994953 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47126) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.166936 [ 3410 ] {c8d9bbbf-3eff-4020-8f97-99a2f0d05a45} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.275993711 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41564) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.167301 [ 3400 ] {5ead47f7-be4b-4d8a-bb29-72dffacc2934} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.039994874 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54402) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.244805 [ 3402 ] {7f58ff5c-ac29-4978-bdac-ee1d0eca2866} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.023994953 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.245755 [ 3402 ] {7f58ff5c-ac29-4978-bdac-ee1d0eca2866} TCPHandler: Client has gone away. 2024.09.24 01:22:04.323123 [ 2663 ] {6a66e478-ce81-47b1-8428-dd7a538db26f} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.179994184 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.327549 [ 2663 ] {6a66e478-ce81-47b1-8428-dd7a538db26f} TCPHandler: Client has gone away. 2024.09.24 01:22:04.476349 [ 1889 ] {c4b3e206-7fe4-4108-b558-d8537693544a} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.275993711 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.477229 [ 1889 ] {c4b3e206-7fe4-4108-b558-d8537693544a} TCPHandler: Client has gone away. 2024.09.24 01:22:04.493857 [ 2661 ] {cb3174ee-2252-4697-9ab0-87e8400880b9} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.235217688 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:04.514731 [ 3410 ] {c8d9bbbf-3eff-4020-8f97-99a2f0d05a45} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.275993711 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.515478 [ 3410 ] {c8d9bbbf-3eff-4020-8f97-99a2f0d05a45} TCPHandler: Client has gone away. 2024.09.24 01:22:04.624262 [ 3400 ] {5ead47f7-be4b-4d8a-bb29-72dffacc2934} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.039994874 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.625151 [ 3400 ] {5ead47f7-be4b-4d8a-bb29-72dffacc2934} TCPHandler: Client has gone away. 2024.09.24 01:22:04.654937 [ 1915 ] {ed9c9d9e-dfd3-4152-8002-6274c24d9cb5} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.331993436 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.655820 [ 1915 ] {ed9c9d9e-dfd3-4152-8002-6274c24d9cb5} TCPHandler: Client has gone away. 2024.09.24 01:22:04.660887 [ 3404 ] {f4c959eb-f4f9-426b-81bb-dd00d86e68d1} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.53599243 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:04.661843 [ 3404 ] {f4c959eb-f4f9-426b-81bb-dd00d86e68d1} TCPHandler: Client has gone away. 2024.09.24 01:22:08.096790 [ 3410 ] {c3893ed5-d4b9-49df-9423-3ccb155ffc36} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.155283893 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54410) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.126909 [ 3400 ] {9a3ba2ad-c56c-48a3-a213-216711e297c1} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.044879291 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53152) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.141529 [ 3410 ] {c3893ed5-d4b9-49df-9423-3ccb155ffc36} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.155283893 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.168600 [ 3400 ] {9a3ba2ad-c56c-48a3-a213-216711e297c1} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.044879291 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.235052 [ 1889 ] {747a5f9b-30a1-4ec8-bd67-bed1e930e11e} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.376974846 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33362) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.250959 [ 3404 ] {905eb525-d234-4c14-8db4-03891026293f} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.353976617 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50010) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.278378 [ 1889 ] {747a5f9b-30a1-4ec8-bd67-bed1e930e11e} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.376974846 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.288174 [ 3404 ] {905eb525-d234-4c14-8db4-03891026293f} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.353976617 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.364478 [ 1915 ] {2c7c8807-18ad-4be3-ad8c-b29f8b6befde} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.517341735 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47130) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.397955 [ 1915 ] {2c7c8807-18ad-4be3-ad8c-b29f8b6befde} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.517341735 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d00f6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 28. ? @ 0x00007fb34c116ac3 in ? 29. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.429624 [ 3402 ] {aaab5d7f-b504-49f9-8a8b-68d619d973d5} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.538731487 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41580) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.560020 [ 2661 ] {6810df1d-61c5-448a-8782-1be2df5376a2} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.119994481 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45906) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:08.652015 [ 3402 ] {aaab5d7f-b504-49f9-8a8b-68d619d973d5} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.538731487 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001ccaa2b9 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e6d0a80 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e6d115c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6d09f6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e6f3a46 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e6f38df in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e6f3855 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e6f3839 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e6f379d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e6f36a7 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e6f3615 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e6f35f5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e6f35d5 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e6f35a0 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a59483 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a61724 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a616d5 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a615fd in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a610e2 in /usr/bin/clickhouse 29. ? @ 0x00007fb34c116ac3 in ? 30. ? @ 0x00007fb34c1a7a04 in ? 2024.09.24 01:22:08.659586 [ 3402 ] {aaab5d7f-b504-49f9-8a8b-68d619d973d5} TCPHandler: Client has gone away. 2024.09.24 01:22:08.690089 [ 1915 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:47130). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013cee00c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ced280 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b2189bc in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e6425a1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 15. ? @ 0x00007fb34c116ac3 in ? 16. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:22:08.707781 [ 3404 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:50010). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013cee00c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ced280 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b2189bc in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e6425a1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e655d65 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x0000000024787579 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x0000000024787dbc in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249dc434 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249d91da in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249d7ede in /usr/bin/clickhouse 15. ? @ 0x00007fb34c116ac3 in ? 16. ? @ 0x00007fb34c1a7a04 in ? (version 23.8.16.41.altinitystable (altinity build)) 2024.09.24 01:22:08.725779 [ 3415 ] {0d7f9783-1e42-4647-afb2-37439ceabf66} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.12799444 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49072) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:08.733592 [ 3396 ] {48b5455a-fb25-4ced-8b67-2644a06e7d2d} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.003995052 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43464) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:08.741217 [ 2663 ] {54724628-e8a1-4746-8d28-d34409a70696} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.027994934 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35944) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:09.584048 [ 3415 ] {0d7f9783-1e42-4647-afb2-37439ceabf66} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.12799444 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:09.588076 [ 3415 ] {0d7f9783-1e42-4647-afb2-37439ceabf66} TCPHandler: Client has gone away. 2024.09.24 01:22:09.797388 [ 2663 ] {54724628-e8a1-4746-8d28-d34409a70696} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.027994934 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:09.802530 [ 2663 ] {54724628-e8a1-4746-8d28-d34409a70696} TCPHandler: Client has gone away. 2024.09.24 01:22:09.878676 [ 2661 ] {6810df1d-61c5-448a-8782-1be2df5376a2} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.119994481 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:09.934444 [ 3396 ] {48b5455a-fb25-4ced-8b67-2644a06e7d2d} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.003995052 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b1a33c2 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b1a338d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x000000002495ef40 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138de6ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b19540a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b87a98c in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b879dd3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b879ce8 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b877e91 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e6eced1 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e6ec980 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e6d19a1 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6d1d37 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e6d2a78 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29d5 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e6d29b5 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e6d2995 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e6d2960 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a5cba3 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a67404 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a673d5 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a673b9 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a6731d in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a67225 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a67195 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a67175 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a67155 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a67120 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013950396 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x000000001394f875 in /usr/bin/clickhouse 2024.09.24 01:22:09.938636 [ 3396 ] {48b5455a-fb25-4ced-8b67-2644a06e7d2d} TCPHandler: Client has gone away. 2024.09.24 01:28:32.938753 [ 775 ] {} Application: Closed all listening sockets. Waiting for 4 outstanding connections. 2024.09.24 01:28:37.846132 [ 775 ] {} Application: Closed connections. But 3 remain. Tip: To increase wait time add to config: 60 2024.09.24 01:28:37.846351 [ 775 ] {} Application: Will shutdown forcefully.