2024.05.01 15:58:00.772851 [ 348 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.05.01 15:58:00.777753 [ 348 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.05.01 15:58:08.871612 [ 348 ] {} <Warning> Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182
2024.05.01 15:58:09.380148 [ 348 ] {} <Warning> KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper.
2024.05.01 15:58:09.381059 [ 348 ] {} <Warning> KeeperLogStore: Removing all changelogs
2024.05.01 15:58:09.383130 [ 348 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.05.01 15:58:09.386571 [ 348 ] {} <Warning> KeeperStateManager: No state was read
2024.05.01 15:58:09.386635 [ 348 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.05.01 15:58:09.386684 [ 348 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.05.01 15:58:09.437657 [ 670 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.05.01 15:58:09.512402 [ 348 ] {} <Warning> Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.05.01 15:58:09.512472 [ 348 ] {} <Warning> Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.05.01 15:58:09.787268 [ 348 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.05.01 15:58:34.480577 [ 348 ] {} <Warning> KeeperLogStore: Changelog is shut down
2024.05.01 15:58:36.979016 [ 844 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.05.01 15:58:36.984317 [ 844 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.05.01 15:58:47.808403 [ 844 ] {} <Warning> Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182
2024.05.01 15:58:48.165054 [ 844 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.05.01 15:58:48.168603 [ 844 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.05.01 15:58:48.168659 [ 844 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.05.01 15:58:48.219550 [ 1078 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.05.01 15:58:48.978764 [ 844 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.05.01 16:02:49.869387 [ 1531 ] {1d602df8-d025-4aca-b221-fcad36975de9} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:50852) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 = (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000036ab79ac in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f19ebc4 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003f0bc003 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f1168b1 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000039d70d9b in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4b974 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4ae1e in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f5788a in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4af03 in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f5f in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.h:35: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x000000003a6ebcbe in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000003a6db834 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x0000000039b23f41 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000039b165b2 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039b09ba7 in /usr/bin/clickhouse
23. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d6a2f9 in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d64e2e in /usr/bin/clickhouse
25. ./build_docker/./contrib/llvm-project/libcxx/include/vector:434: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x0000000039a290f5 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000039a25c6a in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1026: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000003a983fc3 in /usr/bin/clickhouse
28. ./build_docker/./src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x000000003a977782 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x000000003e0542d8 in /usr/bin/clickhouse
30. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse

2024.05.01 16:02:49.870541 [ 1531 ] {1d602df8-d025-4aca-b221-fcad36975de9} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000036ab79ac in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f19ebc4 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003f0bc003 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f1168b1 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000039d70d9b in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4b974 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4ae1e in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f5788a in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4af03 in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f5f in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.h:35: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x000000003a6ebcbe in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000003a6db834 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x0000000039b23f41 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000039b165b2 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039b09ba7 in /usr/bin/clickhouse
23. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d6a2f9 in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d64e2e in /usr/bin/clickhouse
25. ./build_docker/./contrib/llvm-project/libcxx/include/vector:434: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x0000000039a290f5 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000039a25c6a in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1026: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000003a983fc3 in /usr/bin/clickhouse
28. ./build_docker/./src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x000000003a977782 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x000000003e0542d8 in /usr/bin/clickhouse
30. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse

2024.05.01 16:02:50.234694 [ 1531 ] {fe99b1b4-e31e-4739-87b2-d770a53475e2} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:50852) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000036ab79ac in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f19ebc4 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003f0bc003 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f1168b1 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000039d70d9b in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000003a9864a5 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x000000003a977782 in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x000000003e0542d8 in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
17. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
18. ? @ 0x00007f18cb28cac3 in ?
19. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:02:50.235595 [ 1531 ] {fe99b1b4-e31e-4739-87b2-d770a53475e2} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000036ab79ac in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f19ebc4 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003f0bc003 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f1168b1 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000039d70d9b in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000003a9864a5 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x000000003a977782 in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x000000003e0542d8 in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
17. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
18. ? @ 0x00007f18cb28cac3 in ?
19. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:02:50.539678 [ 1531 ] {60d7bf08-77a2-4922-a6d9-667e47977a44} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:50852) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) FROM test.hits WHERE NOT ignore(AdvEngineID);), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:02:50.570482 [ 1531 ] {60d7bf08-77a2-4922-a6d9-667e47977a44} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:02:51.214649 [ 1531 ] {9a902c66-f982-4596-aa05-4b1bf8ab2f8c} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:50852) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID > (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000036ab79ac in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f19ebc4 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003f0bc003 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f1168b1 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000039d70d9b in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4b974 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4ae1e in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f5788a in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4af03 in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.h:35: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x000000003a6ebcbe in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000003a6db834 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x0000000039b23f41 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000039b165b2 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039b09ba7 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d6a2f9 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d64e2e in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/vector:434: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x0000000039a290f5 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000039a25c6a in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:1026: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000003a983fc3 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x000000003a977782 in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x000000003e0542d8 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse

2024.05.01 16:02:51.223968 [ 1531 ] {9a902c66-f982-4596-aa05-4b1bf8ab2f8c} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:65: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x0000000036ab79ac in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f19ebc4 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:0: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x000000003f0bc003 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x000000003f1168b1 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000039d70d9b in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4b974 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4ae1e in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f5788a in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:71: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000039f4af03 in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49cb5 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x0000000039f49f4d in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.h:35: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x000000003a6ebcbe in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000003a6db834 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x0000000039b23f41 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000039b165b2 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039b09ba7 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d6a2f9 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000039d64e2e in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/vector:434: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x0000000039a290f5 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000039a25c6a in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:1026: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000003a983fc3 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1223: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x000000003a977782 in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x000000003e0542d8 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse

2024.05.01 16:02:51.445112 [ 1531 ] {b361ff38-de3b-47f2-9153-7bae16019b94} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:50852) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID < (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:02:51.457302 [ 1531 ] {b361ff38-de3b-47f2-9153-7bae16019b94} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x0000000036ab7e22 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab7555 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:08:55.792361 [ 2672 ] {c5a8d97f-fd6b-42ce-8ba3-475943045617} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108010792 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:38926) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:08:55.792374 [ 2674 ] {12d2e10f-bfd1-4ff2-94fe-7b075111412b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.028010014 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38930) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:08:56.015040 [ 2672 ] {c5a8d97f-fd6b-42ce-8ba3-475943045617} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108010792 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:08:56.255307 [ 2674 ] {12d2e10f-bfd1-4ff2-94fe-7b075111412b} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:38930). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:08:56.255729 [ 2674 ] {12d2e10f-bfd1-4ff2-94fe-7b075111412b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.028010014 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:08:56.256274 [ 2674 ] {12d2e10f-bfd1-4ff2-94fe-7b075111412b} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:08:58.280365 [ 2675 ] {e81d0aa7-ecfe-4e44-a0c8-73fe37c483f1} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.431722661 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38946) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x0000000039f85658 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:197: DB::PipelineExecutor::finalizeExecution() @ 0x000000003e1a7772 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003e1a708a in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1e0d46 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
10. ? @ 0x00007f18cb28cac3 in ?
11. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:08:58.280376 [ 2672 ] {3a4ac2d9-f656-47ca-9456-e54c223c93ea} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208011782 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:38936) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:08:58.280459 [ 2674 ] {52b1d955-789b-4f45-b8e6-0d2e107b7a4b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.084010574 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60180) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:00.440118 [ 2672 ] {3a4ac2d9-f656-47ca-9456-e54c223c93ea} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208011782 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:01.062238 [ 2675 ] {e81d0aa7-ecfe-4e44-a0c8-73fe37c483f1} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.431722661 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:465: DB::QueryStatus::checkTimeLimit() @ 0x0000000039f85658 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:197: DB::PipelineExecutor::finalizeExecution() @ 0x000000003e1a7772 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x000000003e1a708a in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1e0d46 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
10. ? @ 0x00007f18cb28cac3 in ?
11. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:01.062270 [ 2674 ] {52b1d955-789b-4f45-b8e6-0d2e107b7a4b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.084010574 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:01.062321 [ 1883 ] {87542b88-b131-416d-bfae-a8a16ec98aaa} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 576.41 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:38916) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000036ab8397 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab77bc in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:01.102240 [ 2674 ] {52b1d955-789b-4f45-b8e6-0d2e107b7a4b} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:01.112916 [ 2675 ] {e81d0aa7-ecfe-4e44-a0c8-73fe37c483f1} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:01.379864 [ 1883 ] {87542b88-b131-416d-bfae-a8a16ec98aaa} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 576.41 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000036ab8397 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab77bc in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:03.031195 [ 2672 ] {b36f1ace-7d1a-4408-8b63-b6ef5e496b70} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.180011535 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:47632) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:03.335780 [ 2678 ] {5483870c-13f8-459d-a143-b42669aa0971} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.556015213 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48128) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:03.336321 [ 2676 ] {1ea51d67-84d2-4ea1-910d-9cf42b354ee7} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.008009853 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:32768) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:03.349642 [ 2677 ] {d9237c62-1b07-4ac9-8ea2-ec62b1fc8e46} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.47601443 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36090) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:04.431210 [ 2672 ] {b36f1ace-7d1a-4408-8b63-b6ef5e496b70} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.180011535 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:05.606015 [ 2676 ] {1ea51d67-84d2-4ea1-910d-9cf42b354ee7} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.008009853 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:05.677391 [ 2676 ] {1ea51d67-84d2-4ea1-910d-9cf42b354ee7} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:07.103470 [ 2672 ] {e06cf7d9-05f4-4187-8d7b-b22a10c37f8a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.412013827 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:47646) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.183660 [ 2679 ] {81b5f7eb-b0ab-45ba-96ff-ef6776f87e1b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.628015946 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:32770) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.215670 [ 2674 ] {b3d4c96b-3891-4932-9864-3567840a56c7} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.640016063 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48132) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.273186 [ 2677 ] {d9237c62-1b07-4ac9-8ea2-ec62b1fc8e46} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:36090). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:07.273403 [ 2677 ] {d9237c62-1b07-4ac9-8ea2-ec62b1fc8e46} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.47601443 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.282148 [ 2677 ] {d9237c62-1b07-4ac9-8ea2-ec62b1fc8e46} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:07.292231 [ 2680 ] {bc0dca07-bbd0-481d-882b-cf574f7ad76e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.588015555 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55848) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.310291 [ 2675 ] {7b1a2d5a-e847-4127-b285-5e42cb485f34} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.376013479 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36098) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.310833 [ 1883 ] {52a03605-6c5b-4ab3-ac50-d84c461a6c9d} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 575.09 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:38916) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000036ab8397 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab77bc in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.351719 [ 2678 ] {5483870c-13f8-459d-a143-b42669aa0971} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.556015213 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.359386 [ 2678 ] {5483870c-13f8-459d-a143-b42669aa0971} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:07.400656 [ 1883 ] {52a03605-6c5b-4ab3-ac50-d84c461a6c9d} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 575.09 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000036ab8397 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab77bc in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.464691 [ 2679 ] {81b5f7eb-b0ab-45ba-96ff-ef6776f87e1b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.628015946 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.472719 [ 2679 ] {81b5f7eb-b0ab-45ba-96ff-ef6776f87e1b} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:07.492899 [ 2675 ] {7b1a2d5a-e847-4127-b285-5e42cb485f34} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:36098). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:07.493167 [ 2675 ] {7b1a2d5a-e847-4127-b285-5e42cb485f34} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.376013479 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.494464 [ 2675 ] {7b1a2d5a-e847-4127-b285-5e42cb485f34} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:07.497738 [ 2674 ] {b3d4c96b-3891-4932-9864-3567840a56c7} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.640016063 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.499265 [ 2674 ] {b3d4c96b-3891-4932-9864-3567840a56c7} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:07.538236 [ 2680 ] {bc0dca07-bbd0-481d-882b-cf574f7ad76e} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.5]:9000 -> [::ffff:127.0.0.1]:55848). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:07.538490 [ 2680 ] {bc0dca07-bbd0-481d-882b-cf574f7ad76e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.588015555 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:07.557333 [ 2680 ] {bc0dca07-bbd0-481d-882b-cf574f7ad76e} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:07.649589 [ 2672 ] {e06cf7d9-05f4-4187-8d7b-b22a10c37f8a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.412013827 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.588597 [ 2672 ] {db6f7fe0-e6d9-414a-8afc-8bbab68fb6ce} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.484014557 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:47652) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.779596 [ 2676 ] {bc0e37d9-207f-4117-a504-f9cf08fd75ff} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.448014206 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45564) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.839714 [ 2674 ] {99b1304a-64de-4634-9fbd-cf3168a8d93e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.396013695 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34332) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.852976 [ 2675 ] {4dcbdcda-e1c4-4cef-9e89-f5e66d44ead8} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.404013774 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35538) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.860579 [ 2677 ] {fabc7180-9af1-4a4e-a373-dcef59d1cd2e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.328013029 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51564) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.865077 [ 2678 ] {939c043d-6bda-4630-a068-b7e905288b0e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.42401397 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59074) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.912077 [ 1883 ] {67c46370-e1b0-459e-a464-6950d5ac96aa} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 573.89 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:38916) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits PREWHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000036ab8397 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab77bc in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:10.997108 [ 1883 ] {67c46370-e1b0-459e-a464-6950d5ac96aa} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 573.89 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x0000000036ab8397 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x0000000036ab77bc in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a63869 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:11.049676 [ 2672 ] {db6f7fe0-e6d9-414a-8afc-8bbab68fb6ce} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.484014557 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:11.119186 [ 2675 ] {4dcbdcda-e1c4-4cef-9e89-f5e66d44ead8} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.404013774 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:11.120277 [ 2677 ] {fabc7180-9af1-4a4e-a373-dcef59d1cd2e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.328013029 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:11.121587 [ 2675 ] {4dcbdcda-e1c4-4cef-9e89-f5e66d44ead8} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:11.121778 [ 2677 ] {fabc7180-9af1-4a4e-a373-dcef59d1cd2e} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:11.129526 [ 2678 ] {939c043d-6bda-4630-a068-b7e905288b0e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.42401397 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:11.130237 [ 2678 ] {939c043d-6bda-4630-a068-b7e905288b0e} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:11.231522 [ 2674 ] {99b1304a-64de-4634-9fbd-cf3168a8d93e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.396013695 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:11.231979 [ 2674 ] {99b1304a-64de-4634-9fbd-cf3168a8d93e} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:11.276202 [ 2676 ] {bc0e37d9-207f-4117-a504-f9cf08fd75ff} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.5]:9000 -> [::ffff:127.0.0.1]:45564). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:11.276403 [ 2676 ] {bc0e37d9-207f-4117-a504-f9cf08fd75ff} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.448014206 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:11.276751 [ 2676 ] {bc0e37d9-207f-4117-a504-f9cf08fd75ff} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:13.819492 [ 2679 ] {c256d478-bccf-4d2a-8279-d8d3bee291de} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.388013629 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:39974) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.051910 [ 1843 ] {7a85c459-dfef-4348-8b28-df13d22c3608} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.216021767 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45578) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.118191 [ 1883 ] {192392bd-e6b3-47da-ae0a-0c5bc87749d2} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.240012179 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59086) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.200179 [ 2674 ] {82234d61-acc6-408d-ab4d-e44ccf6afbdd} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.03601018 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34336) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.207217 [ 2673 ] {c29e7a6b-7cd6-468e-bed6-fb4e9883ecad} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.492014658 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36510) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.254610 [ 1845 ] {4654f594-b836-4beb-ad55-1d7c3f354535} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.316012925 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35548) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.255456 [ 2672 ] {2fb2b73c-2964-4e40-98ae-928e92770f55} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.476024321 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51580) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.616682 [ 1883 ] {192392bd-e6b3-47da-ae0a-0c5bc87749d2} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:59086). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:14.617288 [ 1883 ] {192392bd-e6b3-47da-ae0a-0c5bc87749d2} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.240012179 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.655624 [ 1883 ] {192392bd-e6b3-47da-ae0a-0c5bc87749d2} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:14.695794 [ 2674 ] {82234d61-acc6-408d-ab4d-e44ccf6afbdd} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.03601018 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.700921 [ 2674 ] {82234d61-acc6-408d-ab4d-e44ccf6afbdd} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:14.706753 [ 2673 ] {c29e7a6b-7cd6-468e-bed6-fb4e9883ecad} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.492014658 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.716086 [ 2673 ] {c29e7a6b-7cd6-468e-bed6-fb4e9883ecad} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:14.716148 [ 2679 ] {c256d478-bccf-4d2a-8279-d8d3bee291de} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.388013629 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.774077 [ 1843 ] {7a85c459-dfef-4348-8b28-df13d22c3608} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.216021767 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.780380 [ 1845 ] {4654f594-b836-4beb-ad55-1d7c3f354535} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:35548). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:14.780612 [ 1845 ] {4654f594-b836-4beb-ad55-1d7c3f354535} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.316012925 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.787472 [ 1843 ] {7a85c459-dfef-4348-8b28-df13d22c3608} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:14.787699 [ 1845 ] {4654f594-b836-4beb-ad55-1d7c3f354535} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:14.826503 [ 2672 ] {2fb2b73c-2964-4e40-98ae-928e92770f55} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:51580). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:14.831525 [ 2672 ] {2fb2b73c-2964-4e40-98ae-928e92770f55} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.476024321 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:14.832455 [ 2672 ] {2fb2b73c-2964-4e40-98ae-928e92770f55} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.165585 [ 2679 ] {3476fcdd-f2a3-4379-bf08-554a446124b3} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.464014398 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:39982) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.208487 [ 1883 ] {09d3e513-ae5a-4869-a536-c8b4f59fbf5e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.412013891 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36518) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.215177 [ 2673 ] {0391c3dc-091b-4923-bdb9-0ce37cf9f730} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.520014951 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51590) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.223264 [ 1845 ] {7068cc4f-9519-4c9b-8b2b-bd0190b947f3} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.432014083 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34352) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.225869 [ 2672 ] {52560df9-1896-4478-8dca-e0464628ded8} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.356013338 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45594) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.272487 [ 2674 ] {38cc3ca0-b920-4609-9110-d07cb07ea4f9} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.352013297 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35554) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.305627 [ 2675 ] {06a325c2-0183-4c91-9b98-c081221d91ea} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.216011962 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59090) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.308266 [ 1843 ] {3ee7b79f-5f46-44d9-9a1b-9fd833ba8fb4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.676016485 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:40256) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.702872 [ 1883 ] {09d3e513-ae5a-4869-a536-c8b4f59fbf5e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.412013891 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.707271 [ 2672 ] {52560df9-1896-4478-8dca-e0464628ded8} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.356013338 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.717202 [ 1883 ] {09d3e513-ae5a-4869-a536-c8b4f59fbf5e} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.719201 [ 2672 ] {52560df9-1896-4478-8dca-e0464628ded8} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.746511 [ 2673 ] {0391c3dc-091b-4923-bdb9-0ce37cf9f730} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.520014951 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.760460 [ 2673 ] {0391c3dc-091b-4923-bdb9-0ce37cf9f730} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.768896 [ 2675 ] {06a325c2-0183-4c91-9b98-c081221d91ea} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:59090). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:18.769133 [ 2675 ] {06a325c2-0183-4c91-9b98-c081221d91ea} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.216011962 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.775178 [ 2675 ] {06a325c2-0183-4c91-9b98-c081221d91ea} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.795400 [ 2674 ] {38cc3ca0-b920-4609-9110-d07cb07ea4f9} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.352013297 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.801685 [ 2674 ] {38cc3ca0-b920-4609-9110-d07cb07ea4f9} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.851075 [ 1843 ] {3ee7b79f-5f46-44d9-9a1b-9fd833ba8fb4} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.8]:9000 -> [::ffff:127.0.0.1]:40256). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:18.851381 [ 1843 ] {3ee7b79f-5f46-44d9-9a1b-9fd833ba8fb4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.676016485 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.865278 [ 1843 ] {3ee7b79f-5f46-44d9-9a1b-9fd833ba8fb4} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.895192 [ 1845 ] {7068cc4f-9519-4c9b-8b2b-bd0190b947f3} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:34352). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:18.895450 [ 1845 ] {7068cc4f-9519-4c9b-8b2b-bd0190b947f3} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.432014083 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:18.896713 [ 1845 ] {7068cc4f-9519-4c9b-8b2b-bd0190b947f3} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:18.938730 [ 2679 ] {3476fcdd-f2a3-4379-bf08-554a446124b3} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.464014398 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.326332 [ 2679 ] {631f0ed3-762c-4f9a-aa0a-bf8556e545ff} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.596025583 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:59566) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.485491 [ 2676 ] {5ce0a628-2543-4708-9f36-490f6a346880} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208011903 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60746) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.508958 [ 2674 ] {aae90a04-b2bb-4f75-9897-53466fd8ff89} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.492024558 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54310) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.542862 [ 1883 ] {cb7b6ade-51e4-45f9-b1e9-545f46f29d39} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.480014585 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48644) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.554673 [ 2675 ] {23a7a282-0723-4460-abe8-2fa6dc9875a1} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.776017503 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47712) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.594229 [ 2673 ] {cb99bbef-79a5-4920-a7d4-9ffd6d97cdc8} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.056010404 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:58270) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.613043 [ 1845 ] {e9f82328-2171-4f35-b505-e4638334d539} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.416013955 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41790) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.656084 [ 1843 ] {b13f2cb4-87eb-40a0-ba69-47fcf9c157c5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.468014467 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37004) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:22.680840 [ 2672 ] {1e237999-7942-430c-bf76-4de7bd404ec9} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.428014073 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59440) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.084985 [ 2675 ] {23a7a282-0723-4460-abe8-2fa6dc9875a1} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.776017503 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.093108 [ 1845 ] {e9f82328-2171-4f35-b505-e4638334d539} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.416013955 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.093303 [ 2675 ] {23a7a282-0723-4460-abe8-2fa6dc9875a1} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:23.093589 [ 1845 ] {e9f82328-2171-4f35-b505-e4638334d539} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:23.093581 [ 1883 ] {cb7b6ade-51e4-45f9-b1e9-545f46f29d39} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.9]:9000 -> [::ffff:127.0.0.1]:48644). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:23.093320 [ 2676 ] {5ce0a628-2543-4708-9f36-490f6a346880} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.7]:9000 -> [::ffff:127.0.0.1]:60746). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:23.093903 [ 2676 ] {5ce0a628-2543-4708-9f36-490f6a346880} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208011903 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.094728 [ 1883 ] {cb7b6ade-51e4-45f9-b1e9-545f46f29d39} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.480014585 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.105572 [ 2676 ] {5ce0a628-2543-4708-9f36-490f6a346880} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:23.105850 [ 1883 ] {cb7b6ade-51e4-45f9-b1e9-545f46f29d39} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:23.117749 [ 2679 ] {631f0ed3-762c-4f9a-aa0a-bf8556e545ff} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.596025583 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.122120 [ 2674 ] {aae90a04-b2bb-4f75-9897-53466fd8ff89} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:54310). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:23.122662 [ 2674 ] {aae90a04-b2bb-4f75-9897-53466fd8ff89} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.492024558 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.127541 [ 2674 ] {aae90a04-b2bb-4f75-9897-53466fd8ff89} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:23.128930 [ 1843 ] {b13f2cb4-87eb-40a0-ba69-47fcf9c157c5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.468014467 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.130503 [ 1843 ] {b13f2cb4-87eb-40a0-ba69-47fcf9c157c5} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:23.131157 [ 2672 ] {1e237999-7942-430c-bf76-4de7bd404ec9} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:59440). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000002109741f in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x00000000210957a2 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x000000003e09973e in /usr/bin/clickhouse
5. ./build_docker/./src/Core/Block.h:25: DB::TCPHandler::sendLogs() @ 0x000000003e072acb in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:680: DB::TCPHandler::runImpl() @ 0x000000003e054caf in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:23.132021 [ 2672 ] {1e237999-7942-430c-bf76-4de7bd404ec9} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.428014073 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.133438 [ 2672 ] {1e237999-7942-430c-bf76-4de7bd404ec9} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:23.138953 [ 2673 ] {cb99bbef-79a5-4920-a7d4-9ffd6d97cdc8} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.056010404 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:23.140348 [ 2673 ] {cb99bbef-79a5-4920-a7d4-9ffd6d97cdc8} <Warning> TCPHandler: Client has gone away.
2024.05.01 16:09:26.573024 [ 2677 ] {cc1dfa62-c805-49d8-9220-0700e047392a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.452014331 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48646) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:26.579403 [ 2675 ] {93883f26-6395-4e34-a1d3-808b18a8c6f4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.660016383 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54320) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:26.624890 [ 2674 ] {f3f196d8-231f-40b4-a397-e03f5f552ca4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.192011765 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41806) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:26.633302 [ 2672 ] {912e5090-0820-439b-bffa-6c4ddaf118fa} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.560015397 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48024) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:26.696217 [ 1845 ] {3cf51a43-82e1-4f33-b090-8f425911ce66} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.648016265 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60752) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:26.709611 [ 1843 ] {bb2c6ec6-f929-427e-874e-d1093485a0ef} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.68401662 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:58286) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.229194 [ 2675 ] {93883f26-6395-4e34-a1d3-808b18a8c6f4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.660016383 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.235870 [ 2677 ] {cc1dfa62-c805-49d8-9220-0700e047392a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.452014331 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.239157 [ 2672 ] {912e5090-0820-439b-bffa-6c4ddaf118fa} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.560015397 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.263237 [ 2674 ] {f3f196d8-231f-40b4-a397-e03f5f552ca4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.192011765 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.288718 [ 2676 ] {96c78947-490c-44b0-86a2-2ebb759bf886} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.676016541 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59446) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.295159 [ 2673 ] {5efc0467-2852-4f23-98c4-e7ab11afe659} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.664016422 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37014) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.302322 [ 1883 ] {9160a33c-0ad7-442e-a75c-20e337aa75c5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.656016344 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47728) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.703994 [ 1883 ] {9160a33c-0ad7-442e-a75c-20e337aa75c5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.656016344 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.732626 [ 1843 ] {bb2c6ec6-f929-427e-874e-d1093485a0ef} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.68401662 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.733930 [ 2673 ] {5efc0467-2852-4f23-98c4-e7ab11afe659} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.664016422 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.741828 [ 1845 ] {3cf51a43-82e1-4f33-b090-8f425911ce66} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.648016265 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.763203 [ 2676 ] {96c78947-490c-44b0-86a2-2ebb759bf886} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.676016541 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:27.788404 [ 2679 ] {7bd0bda7-726a-4b81-9bd2-aeb2961a2fb2} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.572015511 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.11.29.altinitystable (altinity build)) (from [::1]:59578) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:09:28.077499 [ 2676 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:59446). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x0000000021092af9 in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000021091168 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:72: DB::TCPHandler::runImpl() @ 0x000000003e052568 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
11. ? @ 0x00007f18cb28cac3 in ?
12. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:28.078083 [ 2673 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:37014). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x0000000021092af9 in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000021091168 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:72: DB::TCPHandler::runImpl() @ 0x000000003e052568 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
11. ? @ 0x00007f18cb28cac3 in ?
12. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:28.079538 [ 2675 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:54320). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x0000000021092af9 in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x0000000021091168 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:72: DB::TCPHandler::runImpl() @ 0x000000003e052568 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x000000003e09a49c in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000049d19920 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: Poco::Net::TCPServerDispatcher::run() @ 0x0000000049d1a782 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000004a2a0aa6 in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x000000004a29d8ce in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:350: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000004a29a832 in /usr/bin/clickhouse
11. ? @ 0x00007f18cb28cac3 in ?
12. ? @ 0x00007f18cb31da04 in ?
 (version 23.8.11.29.altinitystable (altinity build))
2024.05.01 16:09:28.308304 [ 2679 ] {7bd0bda7-726a-4b81-9bd2-aeb2961a2fb2} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.572015511 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000004a16c8c9 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002083fc93 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x0000000036a66c0c in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x0000000036a65e6c in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:50: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x0000000036a62efc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x000000003e1d51eb in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000003e1a8efd in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/scope_guard.h:48: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000003e1ac2b6 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x0000000020b7b1d3 in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x0000000020b86fb3 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000020b86e5f in /usr/bin/clickhouse
11. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x0000000020b71bdb in /usr/bin/clickhouse
12. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:303: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x0000000020b807eb in /usr/bin/clickhouse
13. ? @ 0x00007f18cb28cac3 in ?
14. ? @ 0x00007f18cb31da04 in ?

2024.05.01 16:15:01.911520 [ 844 ] {} <Warning> Application: Closed all listening sockets. Waiting for 3 outstanding connections.
2024.05.01 16:15:06.818510 [ 844 ] {} <Warning> Application: Closed connections. But 3 remain. Tip: To increase wait time add to config: <shutdown_wait_unfinished>60</shutdown_wait_unfinished>
2024.05.01 16:15:06.818780 [ 844 ] {} <Warning> Application: Will shutdown forcefully.