2024.07.22 01:10:15.086786 [ 402 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.07.22 01:10:15.123486 [ 402 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.07.22 01:10:15.124652 [ 402 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.07.22 01:10:16.281434 [ 402 ] {} <Warning> CloudPlacementInfo: Placement info has not been loaded
2024.07.22 01:10:16.426050 [ 402 ] {} <Warning> KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper.
2024.07.22 01:10:16.427142 [ 402 ] {} <Warning> KeeperLogStore: Removing all changelogs
2024.07.22 01:10:16.428379 [ 402 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.07.22 01:10:16.433550 [ 402 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.07.22 01:10:16.433595 [ 402 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.07.22 01:10:16.485155 [ 993 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.07.22 01:10:16.529717 [ 402 ] {} <Warning> Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.07.22 01:10:16.529816 [ 402 ] {} <Warning> Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.07.22 01:10:16.701615 [ 402 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.07.22 01:10:18.047908 [ 407 ] {40903cbf-8b16-481f-8782-77006e433bed} <Warning> InterpreterCreateQuery: ATTACH TABLE query with full table definition is not recommended: use either ATTACH TABLE hits_v1; to attach existing table or CREATE TABLE hits_v1 <table definition>; to create new table or ATTACH TABLE hits_v1 FROM '/path/to/data/' <table definition>; to create new table and attach data.
2024.07.22 01:10:26.438871 [ 407 ] {448bbfda-00d2-4da3-95f7-debea4cb2df3} <Warning> InterpreterCreateQuery: ATTACH TABLE query with full table definition is not recommended: use either ATTACH TABLE visits_v1; to attach existing table or CREATE TABLE visits_v1 <table definition>; to create new table or ATTACH TABLE visits_v1 FROM '/path/to/data/' <table definition>; to create new table and attach data.
2024.07.22 01:16:41.519861 [ 1665 ] {f1a80ee4-1fc0-4aae-ad48-31f50444b6c3} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:49252). (NETWORK_ERROR) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49252) (comment: 00177_memory_bound_merging.sh) (in query: SELECT `__table1`.`URL` AS `URL`, `__table1`.`EventDate` AS `EventDate`, `__table1`.`URL` AS `max(URL)` FROM `test`.`hits` AS `__table1` WHERE (`__table1`.`CounterID` = 1704509) AND (`__table1`.`UserID` = 4322253409885123546) GROUP BY `__table1`.`CounterID`, `__table1`.`URL`, `__table1`.`EventDate` ORDER BY `__table1`.`URL` ASC, `__table1`.`EventDate` ASC LIMIT _CAST(10, 'UInt64'), _CAST(5, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121851
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001957868d
5. ./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::processOrdinaryQuery() @ 0x0000000019575537
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000019567b8b
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:16:41.588728 [ 1665 ] {f1a80ee4-1fc0-4aae-ad48-31f50444b6c3} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:49252). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121851
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000019585c99
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x000000001957693a
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x00000000195689ca
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:16:41.589036 [ 1665 ] {f1a80ee4-1fc0-4aae-ad48-31f50444b6c3} <Error> TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:49252). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121851
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001957868d
5. ./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::processOrdinaryQuery() @ 0x0000000019575537
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000019567b8b
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:16:41.589677 [ 1665 ] {f1a80ee4-1fc0-4aae-ad48-31f50444b6c3} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:16:53.287245 [ 1527 ] {da267d27-49eb-4ec8-a797-17be2cd7ce16} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60146) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 = (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x00000000162150bd
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:257: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c728d5
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c1ee82
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c4581e
8. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000177e5449
9. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000177e50f0
10. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:2109: DB::(anonymous namespace)::QueryAnalyzer::evaluateScalarSubqueryIfNeeded(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001736e363
11. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x00000000173304f7
12. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6624: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001732ef14
13. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5546: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017347868
14. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x0000000017330076
15. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6624: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001732ef14
16. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017339162
17. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017325e2b
18. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x0000000017322c0f
19. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:0: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001731e07a
20. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x00000000177e4770
21. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000177e1140
22. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> std::__function::__policy_invoker<std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0, std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::InterpreterFactory::Arguments const&) @ 0x00000000177e5a70
23. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001775002b
24. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000017d3b01f
25. ./build_docker/./src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000017d36dd8
26. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: DB::TCPHandler::runImpl() @ 0x0000000019567972
27. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
28. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
29. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
30. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
31. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970

2024.07.22 01:16:53.288077 [ 1527 ] {da267d27-49eb-4ec8-a797-17be2cd7ce16} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x00000000162150bd
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:257: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c728d5
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c1ee82
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c4581e
8. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000177e5449
9. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000177e50f0
10. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:2109: DB::(anonymous namespace)::QueryAnalyzer::evaluateScalarSubqueryIfNeeded(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001736e363
11. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x00000000173304f7
12. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6624: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001732ef14
13. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5546: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017347868
14. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x0000000017330076
15. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6624: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001732ef14
16. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::(anonymous namespace)::QueryAnalyzer::resolveProjectionExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017339162
17. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017325e2b
18. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x0000000017322c0f
19. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:0: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001731e07a
20. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x00000000177e4770
21. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000177e1140
22. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> std::__function::__policy_invoker<std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0, std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::InterpreterFactory::Arguments const&) @ 0x00000000177e5a70
23. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001775002b
24. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000017d3b01f
25. ./build_docker/./src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000017d36dd8
26. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: DB::TCPHandler::runImpl() @ 0x0000000019567972
27. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
28. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
29. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
30. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
31. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970

2024.07.22 01:16:53.569139 [ 1527 ] {09352f49-fd5a-49d0-bb52-8c6ddf6dc434} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60146) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x00000000162150bd
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:257: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c728d5
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c1ee82
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c4581e
8. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000177e5449
9. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000177e50f0
10. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000017d3ba97
11. ./build_docker/./src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000017d36dd8
12. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: DB::TCPHandler::runImpl() @ 0x0000000019567972
13. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
14. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
15. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
16. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
17. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
18. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
19. __tsan_thread_start_func @ 0x000000000700764f
20. ? @ 0x00007f93b7d08ac3
21. ? @ 0x00007f93b7d9a850

2024.07.22 01:16:53.571716 [ 1527 ] {09352f49-fd5a-49d0-bb52-8c6ddf6dc434} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x00000000162150bd
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:257: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c728d5
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c1ee82
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c4581e
8. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000177e5449
9. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000177e50f0
10. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000017d3ba97
11. ./build_docker/./src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000017d36dd8
12. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: DB::TCPHandler::runImpl() @ 0x0000000019567972
13. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
14. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
15. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
16. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
17. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
18. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
19. __tsan_thread_start_func @ 0x000000000700764f
20. ? @ 0x00007f93b7d08ac3
21. ? @ 0x00007f93b7d9a850

2024.07.22 01:16:54.994081 [ 1527 ] {8c204bbc-f871-4320-8df0-4d9bd1c8c1da} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 16.31 million: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TOO_MANY_ROWS) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60146) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) FROM test.hits WHERE NOT ignore(AdvEngineID);), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:16:55.172343 [ 1527 ] {8c204bbc-f871-4320-8df0-4d9bd1c8c1da} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 16.31 million: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:16:55.380489 [ 1527 ] {fcb5fea7-df3d-43b3-9183-61fbf10e846d} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60146) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID > (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x00000000162150bd
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:257: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c728d5
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c1ee82
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c4581e
8. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000177e5449
9. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000177e50f0
10. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:2109: DB::(anonymous namespace)::QueryAnalyzer::evaluateScalarSubqueryIfNeeded(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001736e363
11. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x00000000173304f7
12. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6624: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001732ef14
13. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5546: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017347868
14. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x0000000017330076
15. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x00000000173260ac
16. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x0000000017322c0f
17. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:0: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001731e07a
18. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x00000000177e4770
19. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000177e1140
20. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> std::__function::__policy_invoker<std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0, std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::InterpreterFactory::Arguments const&) @ 0x00000000177e5a70
21. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001775002b
22. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000017d3b01f
23. ./build_docker/./src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000017d36dd8
24. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: DB::TCPHandler::runImpl() @ 0x0000000019567972
25. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
26. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
27. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
28. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
29. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
30. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
31. __tsan_thread_start_func @ 0x000000000700764f

2024.07.22 01:16:55.381267 [ 1527 ] {fcb5fea7-df3d-43b3-9183-61fbf10e846d} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x00000000162150bd
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:257: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c728d5
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c1ee82
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000019c4581e
8. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000177e5449
9. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000177e50f0
10. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:2109: DB::(anonymous namespace)::QueryAnalyzer::evaluateScalarSubqueryIfNeeded(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000001736e363
11. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x00000000173304f7
12. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6624: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000001732ef14
13. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5546: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x0000000017347868
14. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x0000000017330076
15. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x00000000173260ac
16. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x0000000017322c0f
17. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:0: DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000001731e07a
18. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) @ 0x00000000177e4770
19. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:160: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000177e1140
20. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> std::__function::__policy_invoker<std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0, std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::InterpreterFactory::Arguments const&) @ 0x00000000177e5a70
21. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000001775002b
22. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000017d3b01f
23. ./build_docker/./src/Interpreters/executeQuery.cpp:1374: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000017d36dd8
24. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:612: DB::TCPHandler::runImpl() @ 0x0000000019567972
25. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
26. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
27. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
28. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
29. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
30. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
31. __tsan_thread_start_func @ 0x000000000700764f

2024.07.22 01:16:55.874844 [ 1527 ] {645f7dd2-44b0-4e7b-bd12-4528de566754} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_ROWS) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60146) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID < (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:16:55.925789 [ 1527 ] {645f7dd2-44b0-4e7b-bd12-4528de566754} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000016215233
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214de3
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:05.003195 [ 1989 ] {b7569987-014e-47eb-ae72-63c7f69a9073} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.116000866 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60550) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:05.003202 [ 3356 ] {ab48f126-3721-460f-ae49-01fced068dfc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.060000821 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57366) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:05.272145 [ 3356 ] {ab48f126-3721-460f-ae49-01fced068dfc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.060000821 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:05.273792 [ 3356 ] {ab48f126-3721-460f-ae49-01fced068dfc} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:05.307762 [ 1989 ] {b7569987-014e-47eb-ae72-63c7f69a9073} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.116000866 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:07.091945 [ 3356 ] {02a2bc0b-4c4e-48f8-a327-49581cf38e19} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.286631806 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35900) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:07.114216 [ 3358 ] {c29c6f3f-5893-4688-bcff-35996ff912e5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.112000849 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57382) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:07.115803 [ 1989 ] {734eda03-45be-4e97-9c33-78cb60da579e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.164000889 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60564) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:07.133923 [ 3356 ] {02a2bc0b-4c4e-48f8-a327-49581cf38e19} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.286631806 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:07.134600 [ 3356 ] {02a2bc0b-4c4e-48f8-a327-49581cf38e19} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:07.367153 [ 3358 ] {c29c6f3f-5893-4688-bcff-35996ff912e5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.112000849 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:07.368812 [ 3358 ] {c29c6f3f-5893-4688-bcff-35996ff912e5} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:07.379388 [ 1989 ] {734eda03-45be-4e97-9c33-78cb60da579e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.164000889 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:08.995597 [ 3356 ] {95e38409-5a11-4d5b-b54d-a916d8e09a4c} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.09696197 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33188) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:09.046438 [ 3356 ] {95e38409-5a11-4d5b-b54d-a916d8e09a4c} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.09696197 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:09.303759 [ 3358 ] {d3047952-b786-407e-a61a-cafe5e667d0a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.068000804 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57392) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:09.308966 [ 3360 ] {d862b5cb-e81b-4dfa-afe0-94bc38e57b86} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.092000821 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35902) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:09.321957 [ 1989 ] {144e56a5-ab8b-4660-bc85-6e77f5ba5b2f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.040000783 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60570) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:09.395099 [ 3356 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:33188). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:09.533975 [ 3360 ] {d862b5cb-e81b-4dfa-afe0-94bc38e57b86} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.092000821 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:09.534828 [ 3360 ] {d862b5cb-e81b-4dfa-afe0-94bc38e57b86} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:09.542837 [ 3358 ] {d3047952-b786-407e-a61a-cafe5e667d0a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.068000804 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:09.543605 [ 3358 ] {d3047952-b786-407e-a61a-cafe5e667d0a} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:09.607986 [ 1989 ] {144e56a5-ab8b-4660-bc85-6e77f5ba5b2f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.040000783 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.073459 [ 3358 ] {44620f53-553a-44aa-a243-7f487c4bc8cf} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.200117325 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51630) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:198: DB::PipelineExecutor::finalizeExecution() @ 0x0000000019603e08
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603a48
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
10. __tsan_thread_start_func @ 0x000000000700764f
11. ? @ 0x00007f93b7d08ac3
12. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.085717 [ 3362 ] {bb5becd9-1748-45f4-baba-f65beb1da86f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.298028551 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57398) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.132667 [ 3356 ] {1f49514c-c32b-42d2-a004-984d521cf63b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.384120415 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33200) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.162304 [ 3360 ] {bc98d3e1-81f6-4e33-838e-bb589c275f99} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.401516266 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35912) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.186793 [ 1989 ] {7e15ec1d-5d1c-4676-a037-998fdb434fde} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.304000962 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:60580) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.191989 [ 3362 ] {bb5becd9-1748-45f4-baba-f65beb1da86f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.298028551 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.196670 [ 3360 ] {bc98d3e1-81f6-4e33-838e-bb589c275f99} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.401516266 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.199994 [ 3360 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:35912). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:12.200585 [ 3356 ] {1f49514c-c32b-42d2-a004-984d521cf63b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.384120415 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.201268 [ 3356 ] {1f49514c-c32b-42d2-a004-984d521cf63b} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:12.201310 [ 3362 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:57398). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:12.308784 [ 1989 ] {7e15ec1d-5d1c-4676-a037-998fdb434fde} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.304000962 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.320743 [ 3358 ] {44620f53-553a-44aa-a243-7f487c4bc8cf} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.200117325 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:198: DB::PipelineExecutor::finalizeExecution() @ 0x0000000019603e08
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603a48
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
10. __tsan_thread_start_func @ 0x000000000700764f
11. ? @ 0x00007f93b7d08ac3
12. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:12.321202 [ 3358 ] {44620f53-553a-44aa-a243-7f487c4bc8cf} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:14.709211 [ 1989 ] {5f07441d-ead4-41f3-8c3a-fd4532782893} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.184000856 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:55552) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.013492 [ 3362 ] {068ed5e1-04b6-47f8-a243-51c41606f213} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:46972). (NETWORK_ERROR) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46972) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121693
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001957868d
5. ./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::processOrdinaryQuery() @ 0x0000000019575537
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000019567b8b
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.013640 [ 3360 ] {751c7c03-0b35-4067-80bf-d66d17dbb364} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.013793076 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52636) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:198: DB::PipelineExecutor::finalizeExecution() @ 0x0000000019603e08
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603a48
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
10. __tsan_thread_start_func @ 0x000000000700764f
11. ? @ 0x00007f93b7d08ac3
12. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.125944 [ 3362 ] {068ed5e1-04b6-47f8-a243-51c41606f213} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:46972). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121851
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000019585c99
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x000000001957693a
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x00000000195689ca
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:15.126316 [ 3362 ] {068ed5e1-04b6-47f8-a243-51c41606f213} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:46972). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121693
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001957868d
5. ./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::processOrdinaryQuery() @ 0x0000000019575537
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000019567b8b
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.127274 [ 3362 ] {068ed5e1-04b6-47f8-a243-51c41606f213} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:15.198967 [ 1989 ] {5f07441d-ead4-41f3-8c3a-fd4532782893} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.184000856 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.199007 [ 3360 ] {751c7c03-0b35-4067-80bf-d66d17dbb364} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.5]:9000 -> [::ffff:127.0.0.1]:52636). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121693
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000019585c99
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x000000001957693a
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x00000000195689ca
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:15.199055 [ 3356 ] {9ad7416c-99f7-4cb7-a5de-346c9cbead83} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.425478916 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38906) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.199278 [ 3358 ] {bd11fe34-9cd1-4a11-bb8e-5439497cd106} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.507876698 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33016) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.199293 [ 3364 ] {1bdc2891-ab0e-44ac-8cda-fa68f9649a69} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.420516066 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46384) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.199367 [ 3360 ] {751c7c03-0b35-4067-80bf-d66d17dbb364} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.013793076 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:198: DB::PipelineExecutor::finalizeExecution() @ 0x0000000019603e08
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603a48
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
10. __tsan_thread_start_func @ 0x000000000700764f
11. ? @ 0x00007f93b7d08ac3
12. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.200654 [ 3360 ] {751c7c03-0b35-4067-80bf-d66d17dbb364} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:15.232526 [ 3356 ] {9ad7416c-99f7-4cb7-a5de-346c9cbead83} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:38906). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121851
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000019585c99
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x000000001957693a
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x00000000195689ca
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:15.232788 [ 3356 ] {9ad7416c-99f7-4cb7-a5de-346c9cbead83} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.425478916 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.233475 [ 3356 ] {9ad7416c-99f7-4cb7-a5de-346c9cbead83} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:15.237588 [ 3364 ] {1bdc2891-ab0e-44ac-8cda-fa68f9649a69} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:46384). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121851
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000019585c99
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x000000001957693a
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x00000000195689ca
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:15.237886 [ 3364 ] {1bdc2891-ab0e-44ac-8cda-fa68f9649a69} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.420516066 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.238866 [ 3364 ] {1bdc2891-ab0e-44ac-8cda-fa68f9649a69} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:15.239218 [ 3358 ] {bd11fe34-9cd1-4a11-bb8e-5439497cd106} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:33016). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000f1223ad
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000f121851
4. ./src/IO/WriteBuffer.h:65: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000019585c99
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x000000001957693a
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x00000000195689ca
7. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
9. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
12. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:15.239489 [ 3358 ] {bd11fe34-9cd1-4a11-bb8e-5439497cd106} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.507876698 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:15.240290 [ 3358 ] {bd11fe34-9cd1-4a11-bb8e-5439497cd106} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:18.310246 [ 3356 ] {1479c8a1-5b8b-476e-9bc4-8c0d744ab48e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.064894384 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46988) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.492711 [ 3358 ] {fd970bb4-d03e-4b45-bfca-bffb7de21841} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.496001059 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38916) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.492963 [ 3360 ] {cab6790e-250e-4b95-8aba-da5df427beb4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.648001167 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46386) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.493810 [ 3364 ] {1eabdd5e-76d9-4ea4-a127-78f85b393c1f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.520001077 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39138) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.493867 [ 3366 ] {edae5206-eaba-4d21-bbbd-ca1faf3655c1} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.556001101 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52648) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.531762 [ 3356 ] {1479c8a1-5b8b-476e-9bc4-8c0d744ab48e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.064894384 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.606585 [ 3362 ] {b3213cfd-2a9b-47d5-b935-fc202b784523} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.339352774 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33022) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.646244 [ 3362 ] {b3213cfd-2a9b-47d5-b935-fc202b784523} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.339352774 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.658666 [ 1989 ] {94909317-c203-4231-8462-c0159e0959a5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.004001419 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:55554) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.703303 [ 3362 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:33022). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:18.712431 [ 3366 ] {edae5206-eaba-4d21-bbbd-ca1faf3655c1} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.556001101 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.713497 [ 3366 ] {edae5206-eaba-4d21-bbbd-ca1faf3655c1} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:18.713903 [ 3364 ] {1eabdd5e-76d9-4ea4-a127-78f85b393c1f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.520001077 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.714885 [ 3364 ] {1eabdd5e-76d9-4ea4-a127-78f85b393c1f} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:18.717984 [ 3360 ] {cab6790e-250e-4b95-8aba-da5df427beb4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.648001167 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.718799 [ 3360 ] {cab6790e-250e-4b95-8aba-da5df427beb4} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:18.719273 [ 3358 ] {fd970bb4-d03e-4b45-bfca-bffb7de21841} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.496001059 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:18.720004 [ 3358 ] {fd970bb4-d03e-4b45-bfca-bffb7de21841} <Warning> TCPHandler: Client has gone away.
2024.07.22 01:33:19.385824 [ 1989 ] {94909317-c203-4231-8462-c0159e0959a5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.004001419 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:19.653293 [ 2956 ] {9e766135-cb35-4e62-a7d3-6e868bc6155c} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 575.21 MiB: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_BYTES) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:52322) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000016215460
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214f10
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:19.675018 [ 2956 ] {9e766135-cb35-4e62-a7d3-6e868bc6155c} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 575.21 MiB: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000016215460
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214f10
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.306547 [ 3368 ] {ca8242fc-132e-4023-95c5-87f2fc4f712b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.06400073 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52654) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.337498 [ 3366 ] {4db85863-5f65-4fd2-8bbe-567bbc133148} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.056000723 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34198) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.337624 [ 3356 ] {3f9bdbb9-ccf1-40c3-9ec9-f89834289ab9} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016000696 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33028) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.445497 [ 3358 ] {77545e08-e752-41f9-946d-ec5bf24cfabe} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.884123554 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39150) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.460810 [ 3362 ] {24d103ed-d07e-4f83-a815-da255ea477df} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.893446691 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46400) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.474245 [ 3356 ] {3f9bdbb9-ccf1-40c3-9ec9-f89834289ab9} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016000696 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.475663 [ 3366 ] {4db85863-5f65-4fd2-8bbe-567bbc133148} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.056000723 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.479418 [ 3358 ] {77545e08-e752-41f9-946d-ec5bf24cfabe} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.884123554 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.482390 [ 3368 ] {ca8242fc-132e-4023-95c5-87f2fc4f712b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.06400073 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.483792 [ 3360 ] {965f2d0d-7600-47b4-b613-708dad46086f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.913933872 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38922) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.496659 [ 3362 ] {24d103ed-d07e-4f83-a815-da255ea477df} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.893446691 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.497607 [ 3364 ] {103fab55-6943-4348-9786-135038100a0b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.923960389 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46996) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.512827 [ 1989 ] {37daab72-1a57-495f-8d8a-342a81e5c32a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.19200082 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:55562) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.516758 [ 3360 ] {965f2d0d-7600-47b4-b613-708dad46086f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.913933872 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.531278 [ 3364 ] {103fab55-6943-4348-9786-135038100a0b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.923960389 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:22.539161 [ 3364 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:46996). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:22.547859 [ 3362 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:46400). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:22.555118 [ 3360 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:38922). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:22.564072 [ 3356 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:33028). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:22.812426 [ 1989 ] {37daab72-1a57-495f-8d8a-342a81e5c32a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.19200082 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:25.103526 [ 3356 ] {27bd15f8-4048-4593-b753-86d56b990e7f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.214559307 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55882) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:25.360607 [ 3356 ] {27bd15f8-4048-4593-b753-86d56b990e7f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.214559307 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:25.408124 [ 3362 ] {e673c83b-5143-42b5-ba81-cf6dc9fd23fe} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.316000883 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52256) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:25.432215 [ 3360 ] {c394a37c-53de-4a8f-9351-627b5625ed08} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.440000966 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53588) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:25.627551 [ 3362 ] {e673c83b-5143-42b5-ba81-cf6dc9fd23fe} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.316000883 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:25.757115 [ 3360 ] {c394a37c-53de-4a8f-9351-627b5625ed08} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.440000966 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:25.956542 [ 3358 ] {32827f30-eac6-4926-b22b-fbca87c0ac0b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.159671547 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44052) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.004742 [ 3358 ] {32827f30-eac6-4926-b22b-fbca87c0ac0b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.159671547 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.011607 [ 3364 ] {38ec3084-c3de-4604-b56d-fdcd78614bed} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.187902882 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38166) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.022605 [ 3366 ] {13fb5823-c5c4-43ac-83ea-a50840ff452f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.207445311 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47480) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.039218 [ 3368 ] {2afd86dd-4eb8-4710-8dec-acb00d4cee62} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.205636246 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39518) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.042471 [ 3364 ] {38ec3084-c3de-4604-b56d-fdcd78614bed} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.187902882 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.043265 [ 3370 ] {0823a713-25ff-4a6b-a9ef-6d533ad8452d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.1855293 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41066) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.060073 [ 3366 ] {13fb5823-c5c4-43ac-83ea-a50840ff452f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.207445311 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.077223 [ 3368 ] {2afd86dd-4eb8-4710-8dec-acb00d4cee62} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.205636246 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.077382 [ 3370 ] {0823a713-25ff-4a6b-a9ef-6d533ad8452d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.1855293 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.146993 [ 1989 ] {3729b972-bdd9-4c6c-9189-5a52ec10a064} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348000906 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:59034) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.233658 [ 3366 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:47480). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:26.235332 [ 3368 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:39518). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:26.238013 [ 3370 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:41066). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:26.240207 [ 3364 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:38166). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:26.655582 [ 1989 ] {3729b972-bdd9-4c6c-9189-5a52ec10a064} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348000906 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.741493 [ 2956 ] {15489b68-5955-4bed-8a96-0e4a80e6f46b} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 574.30 MiB: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_BYTES) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:52322) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000016215460
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214f10
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:26.758295 [ 2956 ] {15489b68-5955-4bed-8a96-0e4a80e6f46b} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 574.30 MiB: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000016215460
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214f10
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:29.313439 [ 3362 ] {8871042e-d151-499f-b80f-c0cf2d66e9d8} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100000716 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38178) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:29.513495 [ 3368 ] {037a8d5a-ba5a-440b-ade3-fd914ee5a7b4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.12000073 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:39522) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:29.520381 [ 3360 ] {516ea953-b470-430f-b569-ede4b1ed9eed} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108000721 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53596) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:29.545365 [ 3362 ] {8871042e-d151-499f-b80f-c0cf2d66e9d8} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100000716 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.092717 [ 3360 ] {516ea953-b470-430f-b569-ede4b1ed9eed} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108000721 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.145815 [ 3368 ] {037a8d5a-ba5a-440b-ade3-fd914ee5a7b4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.12000073 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.182521 [ 3366 ] {7059f9c6-2550-4b28-b038-a57f244898b9} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.313361006 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47490) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.207974 [ 3358 ] {1cb572a6-482f-4950-85e1-70b06cadab6d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.356274631 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55892) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.212588 [ 3370 ] {54ca0392-4b2a-4985-bdc3-e3a33fc41526} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.35823604 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34544) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.238302 [ 3366 ] {7059f9c6-2550-4b28-b038-a57f244898b9} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.313361006 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.241024 [ 3364 ] {7e9881dd-023b-4faf-9c92-f0d91390010c} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.388019705 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44058) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.260723 [ 3372 ] {beae983c-8b38-4b11-ace1-48c05bdfca38} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.354085134 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52258) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.276010 [ 3358 ] {1cb572a6-482f-4950-85e1-70b06cadab6d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.356274631 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.281573 [ 3370 ] {54ca0392-4b2a-4985-bdc3-e3a33fc41526} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.35823604 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.298668 [ 3364 ] {7e9881dd-023b-4faf-9c92-f0d91390010c} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.388019705 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.304847 [ 3372 ] {beae983c-8b38-4b11-ace1-48c05bdfca38} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.354085134 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.314077 [ 3356 ] {68ccc187-ec8a-4e6e-8985-371bc8eb5f44} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.44995966 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41068) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `__table1`.`SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` AS `__table1` GROUP BY `__table1`.`SearchPhrase` ORDER BY count() DESC LIMIT _CAST(10, 'UInt64')), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.474734 [ 3356 ] {68ccc187-ec8a-4e6e-8985-371bc8eb5f44} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.44995966 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/Interpreters/ProcessList.cpp:482: DB::QueryStatus::checkTimeLimit() @ 0x000000001791b9c0
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:103: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000019603993
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:94: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000019617adc
7. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
9. __tsan_thread_start_func @ 0x000000000700764f
10. ? @ 0x00007f93b7d08ac3
11. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.496527 [ 1989 ] {9bd06258-447b-4366-ab5e-58ddfefe933e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.572001024 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:59038) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:30.581222 [ 3356 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:41068). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:30.583574 [ 3362 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:38178). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000000f11fd1c
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000f11f1d1
4. DB::ReadBuffer::next() @ 0x000000000758643c
5. ./build_docker/./src/Server/TCPHandler.cpp:333: DB::TCPHandler::runImpl() @ 0x0000000019566dca
6. ./build_docker/./src/Server/TCPHandler.cpp:2331: DB::TCPHandler::run() @ 0x0000000019585e08
7. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000001cfd0d43
8. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000001cfd15b2
9. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x000000001d1d26a7
10. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000001d1d0970
11. ./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001d1cee2a
12. __tsan_thread_start_func @ 0x000000000700764f
13. ? @ 0x00007f93b7d08ac3
14. ? @ 0x00007f93b7d9a850
 (version 24.3.5.47.altinitystable (altinity build))
2024.07.22 01:33:30.948371 [ 1989 ] {9bd06258-447b-4366-ab5e-58ddfefe933e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.572001024 seconds, maximum: 1: While executing MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder). (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x00000000161f7646
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:111: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x00000000161f706f
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f5d2c
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:33.010054 [ 2956 ] {74783fbf-b0d8-4e0f-b8d2-a204b01340b1} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 574.96 MiB: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_BYTES) (version 24.3.5.47.altinitystable (altinity build)) (from [::1]:52322) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits PREWHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000016215460
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214f10
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:33:33.077071 [ 2956 ] {74783fbf-b0d8-4e0f-b8d2-a204b01340b1} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 574.96 MiB: While executing MergeTreeSelect(pool: ReadPool, algorithm: Thread). (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x000000001d143683
1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ef8c454
2. ./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000016215460
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000016214f10
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x00000000161f6163
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000019613551
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x00000000196054f0
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:381: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000019605b71
8. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000f06e3af
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000000f07277d
10. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x000000000f0726e2
11. ./base/base/../base/wide_integer_impl.h:810: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f06a68f
12. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f070552
13. __tsan_thread_start_func @ 0x000000000700764f
14. ? @ 0x00007f93b7d08ac3
15. ? @ 0x00007f93b7d9a850

2024.07.22 01:45:25.731874 [ 402 ] {} <Warning> Application: Closed all listening sockets. Waiting for 4 outstanding connections.
2024.07.22 01:45:30.637975 [ 402 ] {} <Warning> Application: Closed connections. But 1 remain. Tip: To increase wait time add to config: <shutdown_wait_unfinished>60</shutdown_wait_unfinished>
2024.07.22 01:45:30.638147 [ 402 ] {} <Warning> Application: Will shutdown forcefully.