2024.05.01 14:29:07.036786 [ 341 ] {} Context: Server was built in debug mode. It will work slowly. 2024.05.01 14:29:07.039564 [ 341 ] {} Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. 2024.05.01 14:29:10.932922 [ 341 ] {} Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182 2024.05.01 14:29:11.623426 [ 341 ] {} KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper. 2024.05.01 14:29:11.624161 [ 341 ] {} KeeperLogStore: Removing all changelogs 2024.05.01 14:29:11.625589 [ 341 ] {} KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647. 2024.05.01 14:29:11.629231 [ 341 ] {} KeeperStateManager: No state was read 2024.05.01 14:29:11.629325 [ 341 ] {} RaftInstance: invalid election timeout lower bound detected, adjusted to 0 2024.05.01 14:29:11.629401 [ 341 ] {} RaftInstance: invalid election timeout upper bound detected, adjusted to 0 2024.05.01 14:29:11.680649 [ 599 ] {} RaftInstance: Election timeout, initiate leader election 2024.05.01 14:29:11.770094 [ 341 ] {} Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist 2024.05.01 14:29:11.770244 [ 341 ] {} Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/ 2024.05.01 14:29:12.125480 [ 341 ] {} Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring. 2024.05.01 14:29:39.197248 [ 602 ] {} RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled 2024.05.01 14:29:39.209671 [ 598 ] {} KeeperLogStore: Raft server is not set in LogStore. 2024.05.01 14:29:39.283326 [ 341 ] {} KeeperLogStore: Changelog is shut down 2024.05.01 14:29:41.139795 [ 774 ] {} Context: Server was built in debug mode. It will work slowly. 2024.05.01 14:29:41.142448 [ 774 ] {} Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. 2024.05.01 14:29:45.098424 [ 774 ] {} Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182 2024.05.01 14:29:45.791971 [ 774 ] {} KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647. 2024.05.01 14:29:45.795941 [ 774 ] {} RaftInstance: invalid election timeout lower bound detected, adjusted to 0 2024.05.01 14:29:45.796024 [ 774 ] {} RaftInstance: invalid election timeout upper bound detected, adjusted to 0 2024.05.01 14:29:45.847344 [ 994 ] {} RaftInstance: Election timeout, initiate leader election 2024.05.01 14:29:47.027631 [ 774 ] {} Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring. 2024.05.01 14:34:24.951845 [ 1695 ] {20b8fc4c-9cfd-42d5-b8eb-58a39b3f9acd} executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:43526) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 = (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8979b9 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee2c6e3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee3b026 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ede828c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee0e81f in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbbd74b in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7fa76 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef61 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc7dda5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8161d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef97 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 31. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 2024.05.01 14:34:24.953000 [ 1695 ] {20b8fc4c-9cfd-42d5-b8eb-58a39b3f9acd} TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8979b9 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee2c6e3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee3b026 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ede828c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee0e81f in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbbd74b in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7fa76 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef61 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc7dda5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8161d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef97 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 31. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 2024.05.01 14:34:25.114521 [ 1695 ] {9e17fe48-fc72-4962-860e-7642aedf62a8} executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:43526) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8979b9 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee2c6e3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee3b026 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ede828c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee0e81f in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbbd74b in /usr/bin/clickhouse 13. /build/src/Interpreters/executeQuery.cpp:1085: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001d0a0db7 in /usr/bin/clickhouse 14. /build/src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x000000001d09c0c4 in /usr/bin/clickhouse 15. /build/src/Server/TCPHandler.cpp:496: DB::TCPHandler::runImpl() @ 0x000000001e5da14d in /usr/bin/clickhouse 16. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 22. ? @ 0x00007f181de03ac3 in ? 23. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:34:25.115432 [ 1695 ] {9e17fe48-fc72-4962-860e-7642aedf62a8} TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8979b9 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee2c6e3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee3b026 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ede828c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee0e81f in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbbd74b in /usr/bin/clickhouse 13. /build/src/Interpreters/executeQuery.cpp:1085: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000001d0a0db7 in /usr/bin/clickhouse 14. /build/src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x000000001d09c0c4 in /usr/bin/clickhouse 15. /build/src/Server/TCPHandler.cpp:496: DB::TCPHandler::runImpl() @ 0x000000001e5da14d in /usr/bin/clickhouse 16. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 17. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 18. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 19. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 20. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 21. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 22. ? @ 0x00007f181de03ac3 in ? 23. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:34:25.373214 [ 1695 ] {d6a667f3-9ac1-43fa-90b8-ac66c917288d} executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:43526) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) FROM test.hits WHERE NOT ignore(AdvEngineID);), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:34:25.380301 [ 1695 ] {d6a667f3-9ac1-43fa-90b8-ac66c917288d} TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:34:25.566078 [ 1695 ] {5c179bcf-7d27-4bce-a092-f58e0f3f44c6} executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:43526) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID > (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8979b9 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee2c6e3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee3b026 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ede828c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee0e81f in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbbd74b in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7fa76 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef61 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc7dda5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8161d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef97 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc7dda5 in /usr/bin/clickhouse 31. /build/src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr&, std::shared_ptr, unsigned long, std::map, std::allocator>>&, std::map, std::allocator>>&, bool, bool) @ 0x000000001cf904c8 in /usr/bin/clickhouse 2024.05.01 14:34:25.567005 [ 1695 ] {5c179bcf-7d27-4bce-a092-f58e0f3f44c6} TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x000000001b8979b9 in /usr/bin/clickhouse 8. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:234: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x000000001ee2c6e3 in /usr/bin/clickhouse 9. /build/src/Processors/QueryPlan/ReadFromMergeTree.cpp:1791: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee3b026 in /usr/bin/clickhouse 10. /build/src/Processors/QueryPlan/ISourceStep.cpp:20: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000001ede828c in /usr/bin/clickhouse 11. /build/src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000001ee0e81f in /usr/bin/clickhouse 12. /build/src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x000000001cbbd74b in /usr/bin/clickhouse 13. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:186: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7fa76 in /usr/bin/clickhouse 14. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:69: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef61 in /usr/bin/clickhouse 15. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 16. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 17. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 18. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 19. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 20. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 21. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc7dda5 in /usr/bin/clickhouse 22. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:321: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc8161d in /usr/bin/clickhouse 23. /build/src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000001cc7ef97 in /usr/bin/clickhouse 24. /build/src/Interpreters/InDepthNodeVisitor.h:71: DB::InDepthNodeVisitor>::doVisit(std::shared_ptr&) @ 0x000000001cc7e0c0 in /usr/bin/clickhouse 25. /build/src/Interpreters/InDepthNodeVisitor.h:61: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2a1 in /usr/bin/clickhouse 26. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 27. /build/src/Interpreters/InDepthNodeVisitor.h:83: void DB::InDepthNodeVisitor>::visitChildren(std::shared_ptr&) @ 0x000000001cc7e346 in /usr/bin/clickhouse 28. /build/src/Interpreters/InDepthNodeVisitor.h:65: void DB::InDepthNodeVisitor>::visitImplMain(std::shared_ptr&) @ 0x000000001cc7e2ae in /usr/bin/clickhouse 29. /build/src/Interpreters/InDepthNodeVisitor.h:53: void DB::InDepthNodeVisitor>::visitImpl(std::shared_ptr&) @ 0x000000001cc7e04a in /usr/bin/clickhouse 30. /build/src/Interpreters/InDepthNodeVisitor.h:33: DB::InDepthNodeVisitor>::visit(std::shared_ptr&) @ 0x000000001cc7dda5 in /usr/bin/clickhouse 31. /build/src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr&, std::shared_ptr, unsigned long, std::map, std::allocator>>&, std::map, std::allocator>>&, bool, bool) @ 0x000000001cf904c8 in /usr/bin/clickhouse 2024.05.01 14:34:25.744875 [ 1695 ] {b04d6e33-d972-4c80-a707-81c180eb81e5} executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:43526) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID < (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:34:25.748608 [ 1695 ] {b04d6e33-d972-4c80-a707-81c180eb81e5} TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, String&&, String&&) @ 0x000000001b897b48 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b89766c in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:43:25.558980 [ 778 ] {ccfc4a81-4852-4da2-addd-0dd8e01262e5} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.560030939 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:35498) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:25.618003 [ 2745 ] {44dd661e-7b1f-45f8-9d70-23588dea652e} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.145602254 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:56260) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:25.798242 [ 2235 ] {724ca166-0a3e-454c-a1b1-8ffeb5c13f33} executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 594.76 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:60012) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b897d68 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b897847 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:43:25.813688 [ 778 ] {ccfc4a81-4852-4da2-addd-0dd8e01262e5} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.560030939 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:25.906151 [ 2745 ] {44dd661e-7b1f-45f8-9d70-23588dea652e} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:56260). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:25.906212 [ 2235 ] {724ca166-0a3e-454c-a1b1-8ffeb5c13f33} TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 594.76 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b897d68 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b897847 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:43:25.906587 [ 2745 ] {44dd661e-7b1f-45f8-9d70-23588dea652e} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.145602254 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:25.907278 [ 2745 ] {44dd661e-7b1f-45f8-9d70-23588dea652e} TCPHandler: Client has gone away. 2024.05.01 14:43:28.414267 [ 2235 ] {bf092aa3-4b88-4d61-bce4-f5e746395f19} executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 595.05 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:60012) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b897d68 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b897847 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:43:28.661610 [ 2235 ] {bf092aa3-4b88-4d61-bce4-f5e746395f19} TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 595.05 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b897d68 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b897847 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:43:28.713056 [ 778 ] {29f0e070-5a91-447c-8c76-f0859bfc524e} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.17323317 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:35514) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:29.180380 [ 2745 ] {1784c444-d385-48cc-95b2-79fab3995b9e} executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:43792). (NETWORK_ERROR) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43792) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2121: DB::TCPHandler::sendProgress() @ 0x000000001e5e249d in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:979: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e231b in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:29.296841 [ 778 ] {29f0e070-5a91-447c-8c76-f0859bfc524e} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.17323317 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:29.296906 [ 2746 ] {e7ffca3b-538d-42e6-9450-00027bb92599} executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:56262). (NETWORK_ERROR) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:56262) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2037: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001e5e4a8f in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:975: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e22aa in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:31.164432 [ 2746 ] {e7ffca3b-538d-42e6-9450-00027bb92599} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:56262). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:31.164882 [ 2746 ] {e7ffca3b-538d-42e6-9450-00027bb92599} TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:56262). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2037: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001e5e4a8f in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:975: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e22aa in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:31.184279 [ 2746 ] {e7ffca3b-538d-42e6-9450-00027bb92599} TCPHandler: Client has gone away. 2024.05.01 14:43:31.277781 [ 2235 ] {c6de2387-7943-48e1-9afe-bd11030dcc2f} executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 584.26 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:60012) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits PREWHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b897d68 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b897847 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:43:31.328066 [ 2235 ] {c6de2387-7943-48e1-9afe-bd11030dcc2f} TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 584.26 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000001b897d68 in /usr/bin/clickhouse 6. /build/src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000001b897847 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86de24 in /usr/bin/clickhouse 8. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 19. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 31. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 2024.05.01 14:43:31.437209 [ 2745 ] {1784c444-d385-48cc-95b2-79fab3995b9e} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:43792). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:31.437730 [ 2745 ] {1784c444-d385-48cc-95b2-79fab3995b9e} TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:43792). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2121: DB::TCPHandler::sendProgress() @ 0x000000001e5e249d in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:979: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e231b in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:31.453823 [ 2745 ] {1784c444-d385-48cc-95b2-79fab3995b9e} TCPHandler: Client has gone away. 2024.05.01 14:43:32.305614 [ 778 ] {f7bf6b8a-dd17-418b-b65b-1f48750613fb} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.189538445 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:42764) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.426428 [ 2753 ] {fd31ef3e-d8f4-4fe2-9677-419d611a6405} executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:39250). (NETWORK_ERROR) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39250) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2037: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001e5e4a8f in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:975: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e22aa in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.431423 [ 2746 ] {fb145a2b-c622-4f17-bcc5-ae2c122e80da} executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:40084). (NETWORK_ERROR) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:40084) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2121: DB::TCPHandler::sendProgress() @ 0x000000001e5e249d in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:979: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e231b in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.439455 [ 1819 ] {2e9d517d-de6c-4187-b484-c126850254e2} executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:59302). (NETWORK_ERROR) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59302) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2121: DB::TCPHandler::sendProgress() @ 0x000000001e5e249d in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:979: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e231b in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.553278 [ 778 ] {f7bf6b8a-dd17-418b-b65b-1f48750613fb} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.189538445 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.677198 [ 2753 ] {fd31ef3e-d8f4-4fe2-9677-419d611a6405} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:39250). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:32.677629 [ 2753 ] {fd31ef3e-d8f4-4fe2-9677-419d611a6405} TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:39250). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2037: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001e5e4a8f in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:975: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e22aa in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.678152 [ 2753 ] {fd31ef3e-d8f4-4fe2-9677-419d611a6405} TCPHandler: Client has gone away. 2024.05.01 14:43:32.684280 [ 2746 ] {fb145a2b-c622-4f17-bcc5-ae2c122e80da} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:40084). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:32.684851 [ 2746 ] {fb145a2b-c622-4f17-bcc5-ae2c122e80da} TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:40084). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2121: DB::TCPHandler::sendProgress() @ 0x000000001e5e249d in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:979: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e231b in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.685642 [ 2746 ] {fb145a2b-c622-4f17-bcc5-ae2c122e80da} TCPHandler: Client has gone away. 2024.05.01 14:43:32.745820 [ 1819 ] {2e9d517d-de6c-4187-b484-c126850254e2} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:59302). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:32.746197 [ 1819 ] {2e9d517d-de6c-4187-b484-c126850254e2} TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:59302). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2121: DB::TCPHandler::sendProgress() @ 0x000000001e5e249d in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:979: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e231b in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:32.746761 [ 1819 ] {2e9d517d-de6c-4187-b484-c126850254e2} TCPHandler: Client has gone away. 2024.05.01 14:43:35.663109 [ 2745 ] {81e8b3f9-d1de-4995-8217-96c86fe5f317} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.00801182 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:42782) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:35.663179 [ 2746 ] {85c29952-89ad-4ed4-9f23-761dc199b130} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.111343069 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35412) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:35.760036 [ 1819 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:59304). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:35.835998 [ 776 ] {1b0a993d-061a-4ec1-992d-4c2fde474388} executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:39264). (NETWORK_ERROR) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39264) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2103: DB::TCPHandler::sendEndOfStream() @ 0x000000001e5e2dc3 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:575: DB::TCPHandler::runImpl() @ 0x000000001e5da5e8 in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 16. ? @ 0x00007f181de03ac3 in ? 17. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:35.836138 [ 2753 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:40094). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:35.840362 [ 776 ] {1b0a993d-061a-4ec1-992d-4c2fde474388} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:39264). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:35.840738 [ 776 ] {1b0a993d-061a-4ec1-992d-4c2fde474388} TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:39264). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2103: DB::TCPHandler::sendEndOfStream() @ 0x000000001e5e2dc3 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:575: DB::TCPHandler::runImpl() @ 0x000000001e5da5e8 in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 16. ? @ 0x00007f181de03ac3 in ? 17. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:35.845161 [ 776 ] {1b0a993d-061a-4ec1-992d-4c2fde474388} TCPHandler: Client has gone away. 2024.05.01 14:43:36.146905 [ 2746 ] {85c29952-89ad-4ed4-9f23-761dc199b130} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.111343069 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:36.147860 [ 2746 ] {85c29952-89ad-4ed4-9f23-761dc199b130} TCPHandler: Client has gone away. 2024.05.01 14:43:36.192082 [ 2745 ] {81e8b3f9-d1de-4995-8217-96c86fe5f317} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.00801182 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:39.386430 [ 2745 ] {51371cf6-2e35-4ae4-af02-170b467b2c6c} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.028012031 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:42790) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:39.473349 [ 1819 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:40108). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:39.569305 [ 776 ] {218151bd-4f9a-414f-aa48-c47689a318be} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.364154689 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59306) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:39.612971 [ 2753 ] {b1489bea-31e2-4138-9132-823d679100f6} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.375623613 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35416) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:39.612472 [ 2746 ] {31c96389-19ac-4ca7-b702-729f013ca614} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.354663307 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33528) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:39.618740 [ 3534 ] {3362ed5f-dfa0-4665-8b26-57d9cddd7efa} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.260380397 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39278) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:39.896257 [ 2745 ] {51371cf6-2e35-4ae4-af02-170b467b2c6c} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.028012031 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:40.340921 [ 3534 ] {3362ed5f-dfa0-4665-8b26-57d9cddd7efa} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.260380397 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:40.352115 [ 3534 ] {3362ed5f-dfa0-4665-8b26-57d9cddd7efa} TCPHandler: Client has gone away. 2024.05.01 14:43:40.483694 [ 776 ] {218151bd-4f9a-414f-aa48-c47689a318be} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.364154689 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:40.488876 [ 776 ] {218151bd-4f9a-414f-aa48-c47689a318be} TCPHandler: Client has gone away. 2024.05.01 14:43:40.758350 [ 2753 ] {b1489bea-31e2-4138-9132-823d679100f6} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.5]:9000 -> [::ffff:127.0.0.1]:35416). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:40.758847 [ 2753 ] {b1489bea-31e2-4138-9132-823d679100f6} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.375623613 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:40.762683 [ 2753 ] {b1489bea-31e2-4138-9132-823d679100f6} TCPHandler: Client has gone away. 2024.05.01 14:43:40.788516 [ 2746 ] {31c96389-19ac-4ca7-b702-729f013ca614} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:33528). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:40.789040 [ 2746 ] {31c96389-19ac-4ca7-b702-729f013ca614} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.354663307 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:40.789910 [ 2746 ] {31c96389-19ac-4ca7-b702-729f013ca614} TCPHandler: Client has gone away. 2024.05.01 14:43:43.224490 [ 2745 ] {f7b406bc-cc6a-409b-8dbf-d94ce5030307} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.052012289 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:42668) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:43.225832 [ 2753 ] {943f3af8-18a1-47be-888d-dbdf2ae76ac2} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.054850811 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52260) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e666ab6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 28. ? @ 0x00007f181de03ac3 in ? 29. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.237959 [ 2753 ] {943f3af8-18a1-47be-888d-dbdf2ae76ac2} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.054850811 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e666ab6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 28. ? @ 0x00007f181de03ac3 in ? 29. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.247919 [ 3534 ] {abef5eba-9310-486e-8163-d00f2561fe99} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.199561804 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50310) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.272204 [ 2753 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:52260). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:43.332135 [ 3534 ] {abef5eba-9310-486e-8163-d00f2561fe99} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.199561804 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.339297 [ 2746 ] {6c8ba394-a541-42b1-98ad-8caa115172d7} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.299812634 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46810) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.355815 [ 3534 ] {abef5eba-9310-486e-8163-d00f2561fe99} TCPHandler: Client has gone away. 2024.05.01 14:43:43.374920 [ 776 ] {7cc8679c-0382-4976-bd08-60bd5a3da0cd} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.354281946 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59920) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.392561 [ 1819 ] {29220581-ef48-41f8-a24e-e8059a685331} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.45714202 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43910) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.402703 [ 3536 ] {c7574210-7dd8-4f16-8bee-272ecdb297c0} executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:55224). (NETWORK_ERROR) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55224) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2037: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001e5e4a8f in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:975: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e22aa in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.667436 [ 3536 ] {c7574210-7dd8-4f16-8bee-272ecdb297c0} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:55224). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:107: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6ba4 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:43.667840 [ 3536 ] {c7574210-7dd8-4f16-8bee-272ecdb297c0} TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:55224). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2037: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001e5e4a8f in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:975: DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x000000001e5e22aa in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:518: DB::TCPHandler::runImpl() @ 0x000000001e5da331 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:43.681109 [ 3536 ] {c7574210-7dd8-4f16-8bee-272ecdb297c0} TCPHandler: Client has gone away. 2024.05.01 14:43:43.893158 [ 2745 ] {f7b406bc-cc6a-409b-8dbf-d94ce5030307} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.052012289 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:44.070135 [ 2746 ] {6c8ba394-a541-42b1-98ad-8caa115172d7} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.299812634 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:44.074899 [ 2746 ] {6c8ba394-a541-42b1-98ad-8caa115172d7} TCPHandler: Client has gone away. 2024.05.01 14:43:44.442103 [ 776 ] {7cc8679c-0382-4976-bd08-60bd5a3da0cd} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:59920). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:44.442714 [ 776 ] {7cc8679c-0382-4976-bd08-60bd5a3da0cd} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.354281946 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:44.448660 [ 776 ] {7cc8679c-0382-4976-bd08-60bd5a3da0cd} TCPHandler: Client has gone away. 2024.05.01 14:43:44.630945 [ 1819 ] {29220581-ef48-41f8-a24e-e8059a685331} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.45714202 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:44.637060 [ 1819 ] {29220581-ef48-41f8-a24e-e8059a685331} TCPHandler: Client has gone away. 2024.05.01 14:43:47.033667 [ 2745 ] {b6670865-751f-42a8-a351-a2024a2f87f9} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.182327224 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59932) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.102276 [ 1819 ] {307755a2-98c4-4bff-b6a0-f851a3e036c1} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.032012032 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:42684) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:47.174462 [ 2745 ] {b6670865-751f-42a8-a351-a2024a2f87f9} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.182327224 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.181878 [ 2753 ] {39b68393-248c-4279-a345-8c5ffc503e46} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.110851819 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46814) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.257726 [ 2745 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:59932). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:47.289013 [ 2753 ] {39b68393-248c-4279-a345-8c5ffc503e46} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.110851819 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.309868 [ 2753 ] {39b68393-248c-4279-a345-8c5ffc503e46} TCPHandler: Client has gone away. 2024.05.01 14:43:47.324429 [ 3538 ] {345df80b-40e5-4fd5-beb9-91c6ae032319} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.423276991 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52262) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.324755 [ 776 ] {d82d005a-1e21-4e50-a648-79f1a9d60d95} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.480953946 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55240) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.324037 [ 3534 ] {41bd6183-9810-4186-b03c-40ef63511df6} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.530646121 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43918) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.340807 [ 3536 ] {20983fa4-e49f-4685-9fea-58ebb5b923f7} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.479147717 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:40450) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.391151 [ 2746 ] {a4f4891f-ee9e-442e-b973-b95380bc993e} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.501304139 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50314) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.450869 [ 776 ] {d82d005a-1e21-4e50-a648-79f1a9d60d95} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.480953946 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.451898 [ 776 ] {d82d005a-1e21-4e50-a648-79f1a9d60d95} TCPHandler: Client has gone away. 2024.05.01 14:43:47.798823 [ 1819 ] {307755a2-98c4-4bff-b6a0-f851a3e036c1} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.032012032 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:47.852221 [ 3538 ] {345df80b-40e5-4fd5-beb9-91c6ae032319} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.423276991 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.866802 [ 3538 ] {345df80b-40e5-4fd5-beb9-91c6ae032319} TCPHandler: Client has gone away. 2024.05.01 14:43:47.989528 [ 3534 ] {41bd6183-9810-4186-b03c-40ef63511df6} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:43918). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:47.994396 [ 3534 ] {41bd6183-9810-4186-b03c-40ef63511df6} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.530646121 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:47.997846 [ 3534 ] {41bd6183-9810-4186-b03c-40ef63511df6} TCPHandler: Client has gone away. 2024.05.01 14:43:48.135715 [ 3536 ] {20983fa4-e49f-4685-9fea-58ebb5b923f7} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.479147717 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:48.139259 [ 3536 ] {20983fa4-e49f-4685-9fea-58ebb5b923f7} TCPHandler: Client has gone away. 2024.05.01 14:43:48.221035 [ 2746 ] {a4f4891f-ee9e-442e-b973-b95380bc993e} TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:50314). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&) @ 0x0000000013ce7588 in /usr/bin/clickhouse 6. /build/src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x0000000013ce6904 in /usr/bin/clickhouse 7. /build/src/IO/WriteBuffer.h:48: DB::WriteBuffer::next() @ 0x000000000b1fc44d in /usr/bin/clickhouse 8. /build/src/Server/TCPHandler.cpp:2073: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000001e5ec635 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2150: DB::TCPHandler::sendLogs() @ 0x000000001e5e2cdf in /usr/bin/clickhouse 10. /build/src/Server/TCPHandler.cpp:672: DB::TCPHandler::runImpl() @ 0x000000001e5db2d7 in /usr/bin/clickhouse 11. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 12. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 13. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 15. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 16. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 17. ? @ 0x00007f181de03ac3 in ? 18. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:48.221415 [ 2746 ] {a4f4891f-ee9e-442e-b973-b95380bc993e} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.501304139 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:48.221971 [ 2746 ] {a4f4891f-ee9e-442e-b973-b95380bc993e} TCPHandler: Client has gone away. 2024.05.01 14:43:50.936898 [ 3536 ] {78a6a3d8-7431-498f-a619-ef2d3cb8c797} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.12674097 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:58756) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e666ab6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 28. ? @ 0x00007f181de03ac3 in ? 29. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:50.960603 [ 3536 ] {78a6a3d8-7431-498f-a619-ef2d3cb8c797} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.12674097 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e666ab6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 28. ? @ 0x00007f181de03ac3 in ? 29. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.057710 [ 776 ] {e193b383-9f37-45a2-b253-f39e6eaf95b6} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.317400508 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45000) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.290774 [ 776 ] {e193b383-9f37-45a2-b253-f39e6eaf95b6} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.317400508 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.367170 [ 3534 ] {083197b9-1ce2-4bf9-acfa-0988d2b4f123} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.712420889 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34728) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.367742 [ 3538 ] {8f9c2f65-bb25-4e95-9f0f-1a2e84167e0a} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.645591223 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35458) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.383787 [ 2746 ] {2cdfed84-6fad-478f-a649-728867eb9bb1} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.687122508 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54098) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.389397 [ 2745 ] {feac45a5-b37a-4aa5-ae40-7d1c8c7aecbe} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.702370664 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57066) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.402483 [ 2753 ] {91e3c677-ebde-40b6-a960-df5408c46efd} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.723227893 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60452) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:51.418402 [ 3540 ] {d720a6e3-c241-486d-aa0c-b7bf26b4cb57} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.660376434 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38428) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:52.028656 [ 2745 ] {feac45a5-b37a-4aa5-ae40-7d1c8c7aecbe} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.702370664 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:52.218205 [ 3534 ] {083197b9-1ce2-4bf9-acfa-0988d2b4f123} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.712420889 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:52.288575 [ 3538 ] {8f9c2f65-bb25-4e95-9f0f-1a2e84167e0a} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.645591223 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:52.312868 [ 2753 ] {91e3c677-ebde-40b6-a960-df5408c46efd} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.723227893 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:52.373080 [ 2746 ] {2cdfed84-6fad-478f-a649-728867eb9bb1} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.687122508 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:52.459357 [ 3540 ] {d720a6e3-c241-486d-aa0c-b7bf26b4cb57} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.660376434 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:52.506364 [ 1819 ] {067a2be3-c770-4b93-9038-c0c5dd2325dc} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.112012941 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:58320) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:52.521087 [ 3540 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:38428). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:52.525042 [ 3536 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:58756). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:52.527058 [ 2746 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:54098). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:52.528113 [ 3534 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:34728). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:52.528748 [ 2745 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:57066). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:53.209840 [ 1819 ] {067a2be3-c770-4b93-9038-c0c5dd2325dc} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.112012941 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:56.467660 [ 2753 ] {1472be1a-0518-40b9-b269-6c9a5f137f3f} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348761669 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57078) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.492817 [ 3536 ] {143449ad-c749-464b-b8c5-a4fd85c20c43} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.200510014 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34740) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.563379 [ 3534 ] {ed31ee63-1057-4a23-b9c0-d5f7eff6faa9} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.398505961 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38432) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e666ab6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 28. ? @ 0x00007f181de03ac3 in ? 29. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.583169 [ 3534 ] {ed31ee63-1057-4a23-b9c0-d5f7eff6faa9} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.398505961 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:102: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e666ab6 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 13. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 16. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 23. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 25. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 28. ? @ 0x00007f181de03ac3 in ? 29. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.591500 [ 2753 ] {1472be1a-0518-40b9-b269-6c9a5f137f3f} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348761669 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.620267 [ 1819 ] {7efe2139-7695-4b2e-bf5c-8229fdec56cb} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016011793 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:58326) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:56.642665 [ 3536 ] {143449ad-c749-464b-b8c5-a4fd85c20c43} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.200510014 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.704628 [ 3534 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:38432). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:56.736290 [ 3536 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:34740). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:56.758998 [ 2753 ] {} ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:57078). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/NetException.h:26: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000013ce5a0c in /usr/bin/clickhouse 6. /build/src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000013ce4c80 in /usr/bin/clickhouse 7. /build/src/IO/ReadBuffer.h:71: DB::ReadBuffer::next() @ 0x000000000b21033c in /usr/bin/clickhouse 8. /build/src/IO/ReadBuffer.h:105: DB::TCPHandler::runImpl() @ 0x000000001e5d8fa1 in /usr/bin/clickhouse 9. /build/src/Server/TCPHandler.cpp:2159: DB::TCPHandler::run() @ 0x000000001e5ec765 in /usr/bin/clickhouse 10. /build/base/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x000000002475c719 in /usr/bin/clickhouse 11. /build/base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x000000002475cf5c in /usr/bin/clickhouse 12. /build/base/poco/Foundation/src/ThreadPool.cpp:188: Poco::PooledThread::run() @ 0x00000000249b1774 in /usr/bin/clickhouse 13. /build/base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000249ae51a in /usr/bin/clickhouse 14. /build/base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000249ad21e in /usr/bin/clickhouse 15. ? @ 0x00007f181de03ac3 in ? 16. ? @ 0x00007f181de94a04 in ? (version 23.8.11.29.altinitystable (altinity build)) 2024.05.01 14:43:56.771684 [ 2745 ] {5f94bb08-0f0b-47bc-bd1e-4c7864f07a5e} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.04001207 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54112) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:56.814845 [ 2746 ] {0c25026c-8b72-41db-8ffb-a02fb81a566c} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.504942627 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35466) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.823066 [ 3542 ] {1e753a74-f1ba-4bca-893c-ca8839a78d46} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.5848977 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60464) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.829589 [ 3540 ] {c006e8dd-fe2c-4330-a3f0-4d8beb31722c} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.48064822 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45014) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.835831 [ 3538 ] {ceceb1e2-3316-44cf-a9cc-7ad3b8793a1c} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.644613928 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:42806) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:56.859601 [ 776 ] {0237b651-0c17-4eb2-895f-c6f797bc6296} executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.616712947 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:58768) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:57.166638 [ 2745 ] {5f94bb08-0f0b-47bc-bd1e-4c7864f07a5e} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.04001207 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:57.179505 [ 2745 ] {5f94bb08-0f0b-47bc-bd1e-4c7864f07a5e} TCPHandler: Client has gone away. 2024.05.01 14:43:57.294048 [ 1819 ] {7efe2139-7695-4b2e-bf5c-8229fdec56cb} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016011793 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list> const&) @ 0x000000001b86dab1 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e683891 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/ExecutionThreadContext.cpp:95: DB::ExecutionThreadContext::executeTask() @ 0x000000001e683340 in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001e668361 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PipelineExecutor.cpp:217: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x000000001e6686f7 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::spawnThreads()::$_0::operator()() const @ 0x000000001e669438 in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669395 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call(DB::PipelineExecutor::spawnThreads()::$_0&) @ 0x000000001e669375 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::operator()[abi:v15000]() @ 0x000000001e669355 in /usr/bin/clickhouse 17. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x000000001e669320 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 20. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x0000000013a54843 in /usr/bin/clickhouse 21. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a5f0a4 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&>(void&&) @ 0x0000000013a5f075 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&, std::__tuple_indices<>) @ 0x0000000013a5f059 in /usr/bin/clickhouse 24. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()&, std::tuple<>&>(void&&, std::tuple<>&) @ 0x0000000013a5efbd in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000013a5eec5 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) @ 0x0000000013a5ee35 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'()&) @ 0x0000000013a5ee15 in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x0000000013a5edf5 in /usr/bin/clickhouse 29. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000013a5edc0 in /usr/bin/clickhouse 30. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 31. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 2024.05.01 14:43:57.380200 [ 3540 ] {c006e8dd-fe2c-4330-a3f0-4d8beb31722c} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.48064822 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:57.380830 [ 2746 ] {0c25026c-8b72-41db-8ffb-a02fb81a566c} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.504942627 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:57.386588 [ 2746 ] {0c25026c-8b72-41db-8ffb-a02fb81a566c} TCPHandler: Client has gone away. 2024.05.01 14:43:57.387083 [ 3540 ] {c006e8dd-fe2c-4330-a3f0-4d8beb31722c} TCPHandler: Client has gone away. 2024.05.01 14:43:57.438027 [ 3542 ] {1e753a74-f1ba-4bca-893c-ca8839a78d46} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.5848977 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:57.442263 [ 3542 ] {1e753a74-f1ba-4bca-893c-ca8839a78d46} TCPHandler: Client has gone away. 2024.05.01 14:43:57.474793 [ 3538 ] {ceceb1e2-3316-44cf-a9cc-7ad3b8793a1c} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.644613928 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:57.477112 [ 3538 ] {ceceb1e2-3316-44cf-a9cc-7ad3b8793a1c} TCPHandler: Client has gone away. 2024.05.01 14:43:57.578587 [ 776 ] {0237b651-0c17-4eb2-895f-c6f797bc6296} TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.616712947 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below): 0. /build/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x000000000b19ad42 in /usr/bin/clickhouse 1. /build/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x000000000b19ad0d in /usr/bin/clickhouse 2. /build/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000024934280 in /usr/bin/clickhouse 3. /build/src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x00000000138d63ae in /usr/bin/clickhouse 4. /build/src/Common/Exception.h:73: DB::Exception::Exception(String&&, int, bool) @ 0x000000000b18cd8a in /usr/bin/clickhouse 5. /build/src/Common/Exception.h:101: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b8705ac in /usr/bin/clickhouse 6. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:110: bool DB::handleOverflowMode(DB::OverflowMode, int, FormatStringHelperImpl::type, std::type_identity::type>, double&&, double&&) @ 0x000000001b86f9f3 in /usr/bin/clickhouse 7. /build/src/QueryPipeline/ExecutionSpeedLimits.cpp:126: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000001b86f908 in /usr/bin/clickhouse 8. /build/src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x000000001cc9a199 in /usr/bin/clickhouse 9. /build/src/Processors/Executors/PipelineExecutor.cpp:185: DB::PipelineExecutor::checkTimeLimit() @ 0x000000001e667440 in /usr/bin/clickhouse 10. /build/src/Processors/Executors/PipelineExecutor.cpp:195: DB::PipelineExecutor::finalizeExecution() @ 0x000000001e667b1c in /usr/bin/clickhouse 11. /build/src/Processors/Executors/PipelineExecutor.cpp:130: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000001e6673b6 in /usr/bin/clickhouse 12. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:84: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) @ 0x000000001e68a406 in /usr/bin/clickhouse 13. /build/src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:113: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x000000001e68a29f in /usr/bin/clickhouse 14. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000](DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x000000001e68a215 in /usr/bin/clickhouse 15. /build/contrib/llvm-project/libcxx/include/tuple:1789: decltype(auto) std::__apply_tuple_impl[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&, std::__tuple_indices<>) @ 0x000000001e68a1f9 in /usr/bin/clickhouse 16. /build/contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15000]&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::tuple<>&) @ 0x000000001e68a15d in /usr/bin/clickhouse 17. /build/src/Common/ThreadPool.h:242: ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x000000001e68a067 in /usr/bin/clickhouse 18. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x000000001e689fd5 in /usr/bin/clickhouse 19. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:480: void std::__invoke_void_return_wrapper::__call::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x000000001e689fb5 in /usr/bin/clickhouse 20. /build/contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() @ 0x000000001e689f95 in /usr/bin/clickhouse 21. /build/contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001e689f60 in /usr/bin/clickhouse 22. /build/contrib/llvm-project/libcxx/include/__functional/function.h:848: std::__function::__policy_func::operator()[abi:v15000]() const @ 0x0000000013948096 in /usr/bin/clickhouse 23. /build/contrib/llvm-project/libcxx/include/__functional/function.h:1187: std::function::operator()() const @ 0x0000000013947575 in /usr/bin/clickhouse 24. /build/src/Common/ThreadPool.cpp:426: ThreadPoolImpl::worker(std::__list_iterator) @ 0x0000000013a51123 in /usr/bin/clickhouse 25. /build/src/Common/ThreadPool.cpp:179: void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()::operator()() const @ 0x0000000013a593c4 in /usr/bin/clickhouse 26. /build/contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval()()) std::__invoke[abi:v15000]::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&) @ 0x0000000013a59375 in /usr/bin/clickhouse 27. /build/contrib/llvm-project/libcxx/include/thread:285: void std::__thread_execute[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(std::tuple::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>&, std::__tuple_indices<>) @ 0x0000000013a5929d in /usr/bin/clickhouse 28. /build/contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x0000000013a58d82 in /usr/bin/clickhouse 29. ? @ 0x00007f181de03ac3 in ? 30. ? @ 0x00007f181de94a04 in ? 2024.05.01 14:43:57.579339 [ 776 ] {0237b651-0c17-4eb2-895f-c6f797bc6296} TCPHandler: Client has gone away. 2024.05.01 14:50:00.456744 [ 774 ] {} Application: Closed all listening sockets. Waiting for 3 outstanding connections. 2024.05.01 14:50:08.140130 [ 997 ] {} RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled 2024.05.01 14:50:08.155484 [ 992 ] {} KeeperLogStore: Raft server is not set in LogStore. 2024.05.01 14:50:08.261216 [ 774 ] {} KeeperLogStore: Changelog is shut down