2024.09.24 09:15:56.365186 [ 346 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.09.24 09:15:56.366765 [ 346 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.09.24 09:15:57.843250 [ 346 ] {} <Warning> Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182
2024.09.24 09:15:58.053140 [ 346 ] {} <Warning> KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper.
2024.09.24 09:15:58.053873 [ 346 ] {} <Warning> KeeperLogStore: Removing all changelogs
2024.09.24 09:15:58.054611 [ 346 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.09.24 09:15:58.058874 [ 346 ] {} <Warning> KeeperStateManager: No state was read
2024.09.24 09:15:58.058911 [ 346 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.09.24 09:15:58.058937 [ 346 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.09.24 09:15:58.109613 [ 572 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.09.24 09:15:58.135010 [ 346 ] {} <Warning> Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.09.24 09:15:58.135050 [ 346 ] {} <Warning> Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.09.24 09:15:58.209910 [ 346 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.09.24 09:16:32.123608 [ 574 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.09.24 09:16:32.134332 [ 569 ] {} <Warning> KeeperLogStore: Raft server is not set in LogStore.
2024.09.24 09:16:32.146309 [ 346 ] {} <Warning> KeeperLogStore: Changelog is shut down
2024.09.24 09:16:32.881340 [ 744 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.09.24 09:16:32.882708 [ 744 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.09.24 09:16:34.345554 [ 744 ] {} <Warning> Context: The setting 'merge_tree_metadata_cache' is enabled. But the feature of 'metadata cache in RocksDB' is experimental and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production. See the corresponding report at https://github.com/ClickHouse/ClickHouse/issues/51182
2024.09.24 09:16:34.560700 [ 744 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.09.24 09:16:34.566153 [ 744 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.09.24 09:16:34.566191 [ 744 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.09.24 09:16:34.617096 [ 963 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.09.24 09:16:34.987907 [ 744 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.09.24 09:18:06.967403 [ 747 ] {df2563e4-b864-4927-8dbf-c628c2cc3ccd} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45064) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 = (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x0000000035cb07b9 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035cec149 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c3fb12 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c91273 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::execute() @ 0x00000000316dc8b9 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031855a65 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031854d1d in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000003185e6bc in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545b4 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x0000000031edc121 in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:1225: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x0000000031ed017f in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:608: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000003150d340 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x00000000315008b1 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:206: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000314f8ab7 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d8338 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d423a in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/vector:580: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x00000000314450f4 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000003144234f in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e429b in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse

2024.09.24 09:18:06.967831 [ 747 ] {df2563e4-b864-4927-8dbf-c628c2cc3ccd} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x0000000035cb07b9 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035cec149 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c3fb12 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c91273 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::execute() @ 0x00000000316dc8b9 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031855a65 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031854d1d in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000003185e6bc in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545b4 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x0000000031edc121 in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:1225: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x0000000031ed017f in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:608: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000003150d340 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x00000000315008b1 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:206: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000314f8ab7 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d8338 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d423a in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/vector:580: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x00000000314450f4 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000003144234f in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e429b in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse

2024.09.24 09:18:07.013348 [ 747 ] {6d94e26f-85b0-44c9-9f4d-faa8aa46c3ba} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45064) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x0000000035cb07b9 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035cec149 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c3fb12 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c91273 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::execute() @ 0x00000000316dc8b9 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e553d in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
17. ? @ 0x00007f9fadd0cac3 in ?
18. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:18:07.013762 [ 747 ] {6d94e26f-85b0-44c9-9f4d-faa8aa46c3ba} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x0000000035cb07b9 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035cec149 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c3fb12 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c91273 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::execute() @ 0x00000000316dc8b9 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e553d in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
17. ? @ 0x00007f9fadd0cac3 in ?
18. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:18:07.121392 [ 747 ] {c290f8de-0e1c-48b5-9f9d-6d97dcde465e} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45064) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) FROM test.hits WHERE NOT ignore(AdvEngineID);), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:18:07.124151 [ 747 ] {c290f8de-0e1c-48b5-9f9d-6d97dcde465e} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:18:07.247633 [ 747 ] {4678d2fc-cec8-41f6-8009-0abd94bd294e} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45064) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID > (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x0000000035cb07b9 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035cec149 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c3fb12 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c91273 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::execute() @ 0x00000000316dc8b9 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031855a65 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031854d1d in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000003185e6bc in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x0000000031edc121 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/TreeRewriter.cpp:1225: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x0000000031ed017f in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:608: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000003150d340 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x00000000315008b1 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:206: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000314f8ab7 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d8338 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d423a in /usr/bin/clickhouse
23. ./build_docker/./contrib/llvm-project/libcxx/include/vector:580: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x00000000314450f4 in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000003144234f in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e429b in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse
27. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
29. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
30. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse

2024.09.24 09:18:07.248408 [ 747 ] {4678d2fc-cec8-41f6-8009-0abd94bd294e} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:233: DB::ReadFromMergeTree::AnalysisResult::checkLimits(DB::Settings const&, DB::SelectQueryInfo const&) const @ 0x0000000035cb07b9 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035cec149 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c3fb12 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c91273 in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::execute() @ 0x00000000316dc8b9 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031855a65 in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x0000000031854d1d in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x000000003185e6bc in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x000000003185425c in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x00000000318545a9 in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/TreeRewriter.cpp:465: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x0000000031edc121 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/TreeRewriter.cpp:1225: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x0000000031ed017f in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:608: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x000000003150d340 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x00000000315008b1 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:206: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000314f8ab7 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d8338 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:0: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x00000000316d423a in /usr/bin/clickhouse
23. ./build_docker/./contrib/llvm-project/libcxx/include/vector:580: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x00000000314450f4 in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000003144234f in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e429b in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse
27. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
29. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
30. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse

2024.09.24 09:18:07.359852 [ 747 ] {d1739afb-3c16-43d5-8b22-24a85f73daff} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45064) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID < (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:18:07.376608 [ 747 ] {d1739afb-3c16-43d5-8b22-24a85f73daff} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x000000002edcf239 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:25: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edce9e5 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:55.235319 [ 3051 ] {74fa2b0b-900f-4611-b79f-a0888c503b75} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.02736783 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41792) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:55.235380 [ 1777 ] {90ac9990-a3e8-4413-813a-e67c8cb0957b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.02878074 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49334) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:55.309590 [ 3051 ] {74fa2b0b-900f-4611-b79f-a0888c503b75} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.02736783 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:55.339742 [ 1777 ] {90ac9990-a3e8-4413-813a-e67c8cb0957b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.02878074 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:55.354950 [ 1778 ] {e3c7d55d-c70b-4949-b32e-947267102f8b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.183524814 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:52440) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:55.389863 [ 1778 ] {e3c7d55d-c70b-4949-b32e-947267102f8b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.183524814 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:55.426501 [ 3051 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:41792). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000001dd602d6 in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000001dd5e719 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:72: DB::TCPHandler::runImpl() @ 0x0000000034ecbd2e in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
10. ? @ 0x00007f9fadd0cac3 in ?
11. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:20:56.822142 [ 1778 ] {ac0be82f-a5de-4daa-9dfc-db552b379d77} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.077128826 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:52442) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:56.882126 [ 1778 ] {ac0be82f-a5de-4daa-9dfc-db552b379d77} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.077128826 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:57.474742 [ 1776 ] {31ba1f54-9a18-4f4b-906b-e0e99c4b8629} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 578.26 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:52428) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000002edcf87e in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edceb9a in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:57.856067 [ 1776 ] {31ba1f54-9a18-4f4b-906b-e0e99c4b8629} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 578.26 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000002edcf87e in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edceb9a in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:58.321059 [ 1778 ] {766f91a8-d9e4-49d6-9e61-567367dfc0bc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.120148665 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:52446) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:58.612829 [ 1778 ] {766f91a8-d9e4-49d6-9e61-567367dfc0bc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.120148665 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:20:59.904383 [ 1776 ] {b2981351-fa19-4ad2-87a4-6c858cbaa297} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 581.50 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:52428) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000002edcf87e in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edceb9a in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:00.035539 [ 1778 ] {9925d072-285d-42d8-8d76-cc20771477b1} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.103588263 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45942) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:00.199158 [ 1778 ] {9925d072-285d-42d8-8d76-cc20771477b1} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.103588263 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:00.288977 [ 1776 ] {b2981351-fa19-4ad2-87a4-6c858cbaa297} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 581.50 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000002edcf87e in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edceb9a in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:00.394305 [ 3051 ] {4e4e2d49-d10c-4153-8bad-36b2279cfc6b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 4.599899183 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49342) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:01.785646 [ 1778 ] {986e224b-af96-4fae-afa6-5e3e5e973dc0} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.162093762 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45958) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:01.899886 [ 1778 ] {986e224b-af96-4fae-afa6-5e3e5e973dc0} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.162093762 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:03.320543 [ 1778 ] {90b149bb-cb9f-41e1-b758-f8788a9ccea3} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.094540262 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45964) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:03.642030 [ 1778 ] {90b149bb-cb9f-41e1-b758-f8788a9ccea3} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.094540262 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:04.150506 [ 1777 ] {dc493d63-2d95-43bf-b326-c0f57cd84f65} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.008029085 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46304) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8e757 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:04.316794 [ 3052 ] {8e0aec42-f9d7-4047-a7e1-c0b29e6f7959} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 8.520601581 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41798) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:05.013979 [ 1778 ] {8ff366ae-9ec7-48af-bfd9-42002ea28b15} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.071815618 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:45966) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:06.517644 [ 1776 ] {a8d656ee-8505-42e5-8d08-6f9903d0aed7} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 596.02 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:52428) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits PREWHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000002edcf87e in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edceb9a in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:07.320894 [ 1778 ] {8ff366ae-9ec7-48af-bfd9-42002ea28b15} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.071815618 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:08.700737 [ 4833 ] {d9a7cae9-b0a6-4117-a8f7-893457aa4b31} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.026415405 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46064) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:08.703485 [ 4833 ] {d9a7cae9-b0a6-4117-a8f7-893457aa4b31} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.026415405 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:08.877585 [ 4834 ] {4cd90843-fb3e-4205-9e64-e73d4203afd0} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.193604372 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34360) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff32fc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
6. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
8. ? @ 0x00007f9fadd0cac3 in ?
9. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:08.877567 [ 1778 ] {6ff834ee-5e24-45ae-bcf2-fa37c340adee} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.256932724 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:54672) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:08.881321 [ 4834 ] {4cd90843-fb3e-4205-9e64-e73d4203afd0} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.193604372 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff32fc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
6. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
8. ? @ 0x00007f9fadd0cac3 in ?
9. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:08.890733 [ 4834 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:34360). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000001dd602d6 in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000001dd5e719 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:72: DB::TCPHandler::runImpl() @ 0x0000000034ecbd2e in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
10. ? @ 0x00007f9fadd0cac3 in ?
11. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:09.490399 [ 3055 ] {f3ee215b-ef1d-4e7b-b51a-a9b34db5f832} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 12.194106977 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51476) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:10.660273 [ 3054 ] {3741132e-6ef9-4415-b6df-af43a2178d95} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 13.364955112 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44918) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:10.740434 [ 3056 ] {e8207f4f-0686-4f52-aa59-62734f47df46} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 13.434965484 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49722) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:11.404528 [ 3053 ] {c9505b34-599f-4292-be37-18613be809ed} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 14.109647289 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49974) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:11.509512 [ 1776 ] {a8d656ee-8505-42e5-8d08-6f9903d0aed7} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 596.02 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x000000002edcf87e in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x000000002edceb9a in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:109: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8eed4 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:11.798028 [ 1778 ] {6ff834ee-5e24-45ae-bcf2-fa37c340adee} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.256932724 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:13.845511 [ 1779 ] {a77ad8ff-4da8-4c64-bfed-daf96925d14a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.154046656 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:54690) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:13.933941 [ 5180 ] {0bdc8f6b-0a5e-4385-a45d-79f0645dcfa0} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.687317276 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59716) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:13.936907 [ 5180 ] {0bdc8f6b-0a5e-4385-a45d-79f0645dcfa0} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.687317276 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:14.607052 [ 5180 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:59716). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x000000001dd602d6 in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:86: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000001dd5e719 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:72: DB::TCPHandler::runImpl() @ 0x0000000034ecbd2e in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
10. ? @ 0x00007f9fadd0cac3 in ?
11. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:14.609473 [ 1779 ] {a77ad8ff-4da8-4c64-bfed-daf96925d14a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.154046656 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:15.017728 [ 3051 ] {4e4e2d49-d10c-4153-8bad-36b2279cfc6b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 4.599899183 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:15.020939 [ 3051 ] {4e4e2d49-d10c-4153-8bad-36b2279cfc6b} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:16.213087 [ 3058 ] {2f3b793c-6005-443c-a421-3c3142be5816} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.80 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49978) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.507034 [ 1088 ] {} <Error> void DB::SystemLog<DB::TraceLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::TraceLogElement]: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.07 GiB (attempt to allocate chunk of 5709632 bytes), maximum: 27.55 GiB. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
6. ./build_docker/./src/Common/Allocator.h:0: Allocator<false, false>::realloc(void*, unsigned long, unsigned long, unsigned long) @ 0x000000001d52b2c7 in /usr/bin/clickhouse
7. void DB::PODArrayBase<8ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::reserveForNextSize<>() @ 0x000000000bdc56d9 in /usr/bin/clickhouse
8. ./build_docker/./src/Common/PODArray.h:322: DB::ColumnVector<unsigned long>::insert(DB::Field const&) @ 0x00000000327b8b80 in /usr/bin/clickhouse
9. ./build_docker/./src/Columns/ColumnArray.cpp:303: DB::ColumnArray::insert(DB::Field const&) @ 0x0000000032347092 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/TraceLog.cpp:59: DB::TraceLogElement::appendToBlock(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&) const @ 0x0000000031e7e66e in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::SystemLog<DB::TraceLogElement>::flushImpl(std::vector<DB::TraceLogElement, std::allocator<DB::TraceLogElement>> const&, unsigned long) @ 0x0000000031ca678c in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/SystemLog.cpp:0: DB::SystemLog<DB::TraceLogElement>::savingThreadFunction() @ 0x0000000031ca4626 in /usr/bin/clickhouse
13. ./build_docker/./src/Common/ThreadPool.h:243: decltype(std::declval<DB::SystemLogBase<DB::TraceLogElement>::startup()::'lambda'()>()()) std::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::SystemLogBase<DB::TraceLogElement>::startup()::'lambda'()>(DB::SystemLogBase<DB::TraceLogElement>::startup()::'lambda'()&&)::'lambda'()&>(DB::SystemLogBase<DB::TraceLogElement>::startup()::'lambda'()&&) @ 0x000000001d9605f9 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
15. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
16. ? @ 0x00007f9fadd0cac3 in ?
17. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:16.590741 [ 3056 ] {e8207f4f-0686-4f52-aa59-62734f47df46} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:49722). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:16.590784 [ 3052 ] {8e0aec42-f9d7-4047-a7e1-c0b29e6f7959} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:41798). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:16.590791 [ 1777 ] {dc493d63-2d95-43bf-b326-c0f57cd84f65} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.008029085 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x000000002ed8e757 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:62: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012cb9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
9. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
10. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
11. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.590836 [ 3054 ] {3741132e-6ef9-4415-b6df-af43a2178d95} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 13.364955112 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.590908 [ 3056 ] {e8207f4f-0686-4f52-aa59-62734f47df46} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 13.434965484 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.590927 [ 3052 ] {8e0aec42-f9d7-4047-a7e1-c0b29e6f7959} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 8.520601581 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.591214 [ 3056 ] {e8207f4f-0686-4f52-aa59-62734f47df46} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:16.591220 [ 1777 ] {dc493d63-2d95-43bf-b326-c0f57cd84f65} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:16.591257 [ 3054 ] {3741132e-6ef9-4415-b6df-af43a2178d95} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:16.591350 [ 3052 ] {8e0aec42-f9d7-4047-a7e1-c0b29e6f7959} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:16.595099 [ 3055 ] {f3ee215b-ef1d-4e7b-b51a-a9b34db5f832} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 12.194106977 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.595437 [ 3055 ] {f3ee215b-ef1d-4e7b-b51a-a9b34db5f832} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:16.609343 [ 3058 ] {2f3b793c-6005-443c-a421-3c3142be5816} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.80 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.609647 [ 3058 ] {2f3b793c-6005-443c-a421-3c3142be5816} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:16.770210 [ 3060 ] {9b91b1cf-e380-484a-94d4-d4a4da5386fc} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.02 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51492) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.770256 [ 3053 ] {c9505b34-599f-4292-be37-18613be809ed} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 14.109647289 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x000000002ed9205d in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x000000002ed91103 in /usr/bin/clickhouse
4. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::finalizeExecution() @ 0x0000000034ff3d54 in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000034ff3440 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:96: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000003501d8a7 in /usr/bin/clickhouse
7. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
8. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
9. ? @ 0x00007f9fadd0cac3 in ?
10. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.771636 [ 1776 ] {c36c4a19-9cf3-42ab-b35b-e940cc9f181d} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.72 GiB (attempt to allocate chunk of 1114397 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: (while reading column UserID): (while reading from part /var/lib/clickhouse/store/5f5/5f55316c-8657-4e60-b714-e6ae3b6e5743/201403_10_18_2/ in table test.hits (5f55316c-8657-4e60-b714-e6ae3b6e5743) located on disk default of type local, from mark 0 with max_rows_to_read = 19001): While executing MergeTreeThread. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:54706) (comment: 00088_global_in_one_shard_and_rows_before_limit.sql) (in query: SELECT EventDate, count() FROM remote('127.0.0.1', test.hits) WHERE UserID GLOBAL IN (SELECT UserID FROM test.hits) GROUP BY EventDate ORDER BY EventDate LIMIT 5 FORMAT JSONCompact;), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<false, false>::alloc(unsigned long, unsigned long) @ 0x000000001d52966f in /usr/bin/clickhouse
9. void DB::PODArrayBase<8ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::resize<>(unsigned long) @ 0x000000000bd861c5 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/PODArray.h:321: DB::SerializationNumber<unsigned long>::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0x000000002f211566 in /usr/bin/clickhouse
11. ./build_docker/./src/DataTypes/Serializations/ISerialization.cpp:0: DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000002f1387fe in /usr/bin/clickhouse
12. ./build_docker/./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:196: DB::MergeTreeReaderWide::readData(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, bool, unsigned long, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&, bool) @ 0x000000003443c4ea in /usr/bin/clickhouse
13. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderWide.cpp:159: DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x000000003443ae0c in /usr/bin/clickhouse
14. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:107: DB::MergeTreeRangeReader::DelayedStream::finalize(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000034481a8b in /usr/bin/clickhouse
15. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::startReadingChain(unsigned long, DB::MarkRanges&) @ 0x000000003449f7ea in /usr/bin/clickhouse
16. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::read(unsigned long, DB::MarkRanges&) @ 0x000000003449a2a1 in /usr/bin/clickhouse
17. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:398: DB::IMergeTreeSelectAlgorithm::readFromPartImpl() @ 0x0000000034475d99 in /usr/bin/clickhouse
18. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:441: DB::IMergeTreeSelectAlgorithm::readFromPart() @ 0x00000000344775c4 in /usr/bin/clickhouse
19. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:0: DB::IMergeTreeSelectAlgorithm::read() @ 0x000000003446fc60 in /usr/bin/clickhouse
20. ./build_docker/./src/Storages/MergeTree/MergeTreeSource.cpp:181: DB::MergeTreeSource::tryGenerate() @ 0x0000000035d28b78 in /usr/bin/clickhouse
21. ./build_docker/./contrib/llvm-project/libcxx/include/optional:344: DB::ISource::work() @ 0x0000000034fcc465 in /usr/bin/clickhouse
22. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
23. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
25. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
26. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
27. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
28. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
29. ? @ 0x00007f9fadd0cac3 in ?
30. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.772043 [ 3919 ] {a3e278a1-fbf9-4bae-8aac-5e9bc186abe9} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.10 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53936) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.772492 [ 3918 ] {ae963eb4-d731-4110-a994-19ea5c333535} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.74 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44932) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.943017 [ 4833 ] {35240d2e-542d-4d1a-82f9-d65271c04fe7} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.66 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43938) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.943477 [ 5183 ] {beb75c10-c5e2-4092-bc17-47d2aee0689f} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.21 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47108) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:16.944628 [ 3053 ] {c9505b34-599f-4292-be37-18613be809ed} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.156214 [ 3057 ] {ca54a6b2-91ce-4c85-9c77-fffcc87a89b5} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.33 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49734) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.160226 [ 3919 ] {a3e278a1-fbf9-4bae-8aac-5e9bc186abe9} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.10 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.159515 [ 3060 ] {9b91b1cf-e380-484a-94d4-d4a4da5386fc} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.02 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.161918 [ 1776 ] {c36c4a19-9cf3-42ab-b35b-e940cc9f181d} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.72 GiB (attempt to allocate chunk of 1114397 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: (while reading column UserID): (while reading from part /var/lib/clickhouse/store/5f5/5f55316c-8657-4e60-b714-e6ae3b6e5743/201403_10_18_2/ in table test.hits (5f55316c-8657-4e60-b714-e6ae3b6e5743) located on disk default of type local, from mark 0 with max_rows_to_read = 19001): While executing MergeTreeThread. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<false, false>::alloc(unsigned long, unsigned long) @ 0x000000001d52966f in /usr/bin/clickhouse
9. void DB::PODArrayBase<8ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::resize<>(unsigned long) @ 0x000000000bd861c5 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/PODArray.h:321: DB::SerializationNumber<unsigned long>::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0x000000002f211566 in /usr/bin/clickhouse
11. ./build_docker/./src/DataTypes/Serializations/ISerialization.cpp:0: DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000002f1387fe in /usr/bin/clickhouse
12. ./build_docker/./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:196: DB::MergeTreeReaderWide::readData(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, bool, unsigned long, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&, bool) @ 0x000000003443c4ea in /usr/bin/clickhouse
13. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderWide.cpp:159: DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x000000003443ae0c in /usr/bin/clickhouse
14. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:107: DB::MergeTreeRangeReader::DelayedStream::finalize(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000034481a8b in /usr/bin/clickhouse
15. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::startReadingChain(unsigned long, DB::MarkRanges&) @ 0x000000003449f7ea in /usr/bin/clickhouse
16. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::read(unsigned long, DB::MarkRanges&) @ 0x000000003449a2a1 in /usr/bin/clickhouse
17. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:398: DB::IMergeTreeSelectAlgorithm::readFromPartImpl() @ 0x0000000034475d99 in /usr/bin/clickhouse
18. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:441: DB::IMergeTreeSelectAlgorithm::readFromPart() @ 0x00000000344775c4 in /usr/bin/clickhouse
19. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:0: DB::IMergeTreeSelectAlgorithm::read() @ 0x000000003446fc60 in /usr/bin/clickhouse
20. ./build_docker/./src/Storages/MergeTree/MergeTreeSource.cpp:181: DB::MergeTreeSource::tryGenerate() @ 0x0000000035d28b78 in /usr/bin/clickhouse
21. ./build_docker/./contrib/llvm-project/libcxx/include/optional:344: DB::ISource::work() @ 0x0000000034fcc465 in /usr/bin/clickhouse
22. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
23. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
25. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
26. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
27. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
28. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
29. ? @ 0x00007f9fadd0cac3 in ?
30. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.161681 [ 3918 ] {ae963eb4-d731-4110-a994-19ea5c333535} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.74 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.164420 [ 5185 ] {f42c5a73-1a26-4be2-950e-d607ac6c6f9f} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.77 GiB (attempt to allocate chunk of 1050209 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59926) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<wide::integer<128ul, unsigned int>, StringHashMapCell<wide::integer<128ul, unsigned int>, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095fb1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095886 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.166711 [ 4530 ] {fcf891c0-032d-45ca-b18f-9de058a4b71e} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.25 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57422) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.166736 [ 4528 ] {bc8993e6-c13e-456f-a737-49654a72c6d3} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.18 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51510) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.171841 [ 3926 ] {f4c557dd-7035-42b3-8c6b-3cc7ec168fcc} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.67 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44938) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.162844 [ 4833 ] {35240d2e-542d-4d1a-82f9-d65271c04fe7} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.66 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.250576 [ 5183 ] {beb75c10-c5e2-4092-bc17-47d2aee0689f} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.21 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.277583 [ 1778 ] {af48bfb0-7942-4f3a-a580-43d97bbe94c3} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.55 GiB (attempt to allocate chunk of 2883584 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: (while reading column URL): (while reading from part /var/lib/clickhouse/store/5f5/5f55316c-8657-4e60-b714-e6ae3b6e5743/201403_10_18_2/ in table test.hits (5f55316c-8657-4e60-b714-e6ae3b6e5743) located on disk default of type local, from mark 40 with max_rows_to_read = 8192): While executing MergeTreeThread. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::1]:54676) (comment: 00089_position_functions_with_non_constant_arg.sql) (in query: SELECT count() FROM test.hits WHERE position(URL, 'metrika') != position(URL, materialize('metrika'));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:0: Allocator<false, false>::realloc(void*, unsigned long, unsigned long, unsigned long) @ 0x000000001d52b2c7 in /usr/bin/clickhouse
9. void DB::PODArrayBase<1ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::resize<>(unsigned long) @ 0x000000000bdbee29 in /usr/bin/clickhouse
10. ./build_docker/./src/DataTypes/Serializations/SerializationString.cpp:176: void DB::deserializeBinarySSE2<4>(DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::ReadBuffer&, unsigned long) @ 0x000000002f275ec4 in /usr/bin/clickhouse
11. ./build_docker/./src/DataTypes/Serializations/SerializationString.cpp:0: DB::SerializationString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0x000000002f274736 in /usr/bin/clickhouse
12. ./build_docker/./src/DataTypes/Serializations/ISerialization.cpp:0: DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000002f1387fe in /usr/bin/clickhouse
13. ./build_docker/./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:196: DB::MergeTreeReaderWide::readData(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, bool, unsigned long, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&, bool) @ 0x000000003443c4ea in /usr/bin/clickhouse
14. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderWide.cpp:159: DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x000000003443ae0c in /usr/bin/clickhouse
15. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:107: DB::MergeTreeRangeReader::DelayedStream::finalize(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000034481a8b in /usr/bin/clickhouse
16. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::startReadingChain(unsigned long, DB::MarkRanges&) @ 0x000000003449f7ea in /usr/bin/clickhouse
17. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::read(unsigned long, DB::MarkRanges&) @ 0x000000003449a2a1 in /usr/bin/clickhouse
18. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:398: DB::IMergeTreeSelectAlgorithm::readFromPartImpl() @ 0x0000000034475d99 in /usr/bin/clickhouse
19. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:441: DB::IMergeTreeSelectAlgorithm::readFromPart() @ 0x00000000344775c4 in /usr/bin/clickhouse
20. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:0: DB::IMergeTreeSelectAlgorithm::read() @ 0x000000003446fc60 in /usr/bin/clickhouse
21. ./build_docker/./src/Storages/MergeTree/MergeTreeSource.cpp:181: DB::MergeTreeSource::tryGenerate() @ 0x0000000035d28b78 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/optional:344: DB::ISource::work() @ 0x0000000034fcc465 in /usr/bin/clickhouse
23. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
26. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
27. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
28. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
29. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
30. ? @ 0x00007f9fadd0cac3 in ?
31. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278304 [ 3923 ] {45cccf4f-87ba-4b1b-afc9-b2295e761997} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.93 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49980) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278433 [ 3925 ] {8c531945-1368-4f3a-98fa-694099fa68e6} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.83 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37986) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278438 [ 4834 ] {8d94b4c2-b4e2-4a9f-9e0d-2b9d12e056d9} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.91 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41856) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278525 [ 3924 ] {3a79b3df-0f6c-4cc6-a5bd-a3c65cfd1ede} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.99 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49738) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278588 [ 3927 ] {e467c5e8-d6e3-4331-937c-86f6fd7949e0} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.81 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51496) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278700 [ 5179 ] {a89cb76a-f84e-4b97-8efb-498d466f64d4} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.02 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:46080) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278584 [ 4180 ] {2657b4bb-c61f-4a75-96b0-93bd938d1259} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.36 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44946) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.278715 [ 4176 ] {69fca91e-4b0e-471e-84f6-34d2cfec7c30} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.08 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49750) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.279020 [ 4181 ] {f5c52405-5dcb-43d8-b0b3-64b838755d6c} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.24 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51506) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.279030 [ 4534 ] {ff29cee3-3302-488d-a9dc-658c4646b9af} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.29 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44948) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.279037 [ 5184 ] {50a834f6-c377-429a-b411-221e89a3666a} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.95 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34290) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.279619 [ 4182 ] {72586ce3-e4c1-4539-af43-db9ffd65f74d} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.35 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33152) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.280350 [ 3060 ] {9b91b1cf-e380-484a-94d4-d4a4da5386fc} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.280384 [ 3919 ] {a3e278a1-fbf9-4bae-8aac-5e9bc186abe9} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.503948 [ 5182 ] {9cecccbf-bd27-4b87-a417-369437478294} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.34 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34788) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504010 [ 5186 ] {ab303059-a412-4925-b67e-d2099ee52191} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.78 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34374) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504076 [ 4533 ] {fa811a6c-a9a5-44a6-88bc-119cc6d4b433} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.64 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49996) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504202 [ 4837 ] {08329455-d687-4a70-86a1-b1dc87f080e4} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.78 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47098) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504340 [ 4529 ] {3e2bb08b-c604-40f0-b972-b1a72180b7b5} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.19 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49760) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504257 [ 4840 ] {54c3aed1-dfc1-4293-a26f-a9bce5f909e4} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.63 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59710) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504087 [ 4528 ] {bc8993e6-c13e-456f-a737-49654a72c6d3} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.18 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504613 [ 4535 ] {3f3f997b-01cd-40ce-8d76-04cc0c9aa898} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.79 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53946) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504802 [ 4532 ] {fd65b6fb-21ee-4971-83e6-6c9eb36d8c60} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.32 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33164) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504744 [ 4838 ] {be620a69-658e-4120-ab8d-842db277c827} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.57 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43934) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504765 [ 4531 ] {e125125c-aa5d-45e5-ae4b-01a5fae56805} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.74 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38012) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504971 [ 5183 ] {beb75c10-c5e2-4092-bc17-47d2aee0689f} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.504906 [ 4179 ] {e2fa0c66-49dd-49b2-883c-92594962b9b1} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.96 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49988) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504930 [ 4833 ] {35240d2e-542d-4d1a-82f9-d65271c04fe7} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.504907 [ 4177 ] {13e8b5c5-2836-46fa-8f5d-5b0832d8b770} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.09 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53942) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504933 [ 1778 ] {af48bfb0-7942-4f3a-a580-43d97bbe94c3} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.55 GiB (attempt to allocate chunk of 2883584 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: (while reading column URL): (while reading from part /var/lib/clickhouse/store/5f5/5f55316c-8657-4e60-b714-e6ae3b6e5743/201403_10_18_2/ in table test.hits (5f55316c-8657-4e60-b714-e6ae3b6e5743) located on disk default of type local, from mark 40 with max_rows_to_read = 8192): While executing MergeTreeThread. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:0: Allocator<false, false>::realloc(void*, unsigned long, unsigned long, unsigned long) @ 0x000000001d52b2c7 in /usr/bin/clickhouse
9. void DB::PODArrayBase<1ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::resize<>(unsigned long) @ 0x000000000bdbee29 in /usr/bin/clickhouse
10. ./build_docker/./src/DataTypes/Serializations/SerializationString.cpp:176: void DB::deserializeBinarySSE2<4>(DB::PODArray<char8_t, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, DB::ReadBuffer&, unsigned long) @ 0x000000002f275ec4 in /usr/bin/clickhouse
11. ./build_docker/./src/DataTypes/Serializations/SerializationString.cpp:0: DB::SerializationString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0x000000002f274736 in /usr/bin/clickhouse
12. ./build_docker/./src/DataTypes/Serializations/ISerialization.cpp:0: DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x000000002f1387fe in /usr/bin/clickhouse
13. ./build_docker/./contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:196: DB::MergeTreeReaderWide::readData(DB::NameAndTypePair const&, std::shared_ptr<DB::ISerialization const> const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, bool, unsigned long, unsigned long, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>&, bool) @ 0x000000003443c4ea in /usr/bin/clickhouse
14. ./build_docker/./src/Storages/MergeTree/MergeTreeReaderWide.cpp:159: DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x000000003443ae0c in /usr/bin/clickhouse
15. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:107: DB::MergeTreeRangeReader::DelayedStream::finalize(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000034481a8b in /usr/bin/clickhouse
16. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::startReadingChain(unsigned long, DB::MarkRanges&) @ 0x000000003449f7ea in /usr/bin/clickhouse
17. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:0: DB::MergeTreeRangeReader::read(unsigned long, DB::MarkRanges&) @ 0x000000003449a2a1 in /usr/bin/clickhouse
18. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:398: DB::IMergeTreeSelectAlgorithm::readFromPartImpl() @ 0x0000000034475d99 in /usr/bin/clickhouse
19. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:441: DB::IMergeTreeSelectAlgorithm::readFromPart() @ 0x00000000344775c4 in /usr/bin/clickhouse
20. ./build_docker/./src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:0: DB::IMergeTreeSelectAlgorithm::read() @ 0x000000003446fc60 in /usr/bin/clickhouse
21. ./build_docker/./src/Storages/MergeTree/MergeTreeSource.cpp:181: DB::MergeTreeSource::tryGenerate() @ 0x0000000035d28b78 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/optional:344: DB::ISource::work() @ 0x0000000034fcc465 in /usr/bin/clickhouse
23. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
26. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
27. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
28. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
29. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
30. ? @ 0x00007f9fadd0cac3 in ?
31. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.504871 [ 3918 ] {ae963eb4-d731-4110-a994-19ea5c333535} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.741956 [ 5182 ] {9cecccbf-bd27-4b87-a417-369437478294} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.34 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.742508 [ 4834 ] {8d94b4c2-b4e2-4a9f-9e0d-2b9d12e056d9} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.91 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.750278 [ 4840 ] {54c3aed1-dfc1-4293-a26f-a9bce5f909e4} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.63 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.750729 [ 4531 ] {e125125c-aa5d-45e5-ae4b-01a5fae56805} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.74 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.753968 [ 5185 ] {f42c5a73-1a26-4be2-950e-d607ac6c6f9f} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.77 GiB (attempt to allocate chunk of 1050209 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<wide::integer<128ul, unsigned int>, StringHashMapCell<wide::integer<128ul, unsigned int>, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095fb1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095886 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.760291 [ 4832 ] {592a98a5-2565-461d-962e-b8197c4fe417} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.80 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:41852) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.777717 [ 4528 ] {bc8993e6-c13e-456f-a737-49654a72c6d3} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.778652 [ 4832 ] {592a98a5-2565-461d-962e-b8197c4fe417} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.80 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.786879 [ 3925 ] {8c531945-1368-4f3a-98fa-694099fa68e6} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.83 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.790228 [ 5184 ] {50a834f6-c377-429a-b411-221e89a3666a} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.10]:9000 -> [::ffff:127.0.0.1]:34290). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:17.818589 [ 4840 ] {54c3aed1-dfc1-4293-a26f-a9bce5f909e4} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.818623 [ 4834 ] {8d94b4c2-b4e2-4a9f-9e0d-2b9d12e056d9} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.821163 [ 5182 ] {9cecccbf-bd27-4b87-a417-369437478294} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.840698 [ 4531 ] {e125125c-aa5d-45e5-ae4b-01a5fae56805} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.853507 [ 4534 ] {ff29cee3-3302-488d-a9dc-658c4646b9af} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.29 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.853538 [ 4177 ] {13e8b5c5-2836-46fa-8f5d-5b0832d8b770} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.09 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.864374 [ 5185 ] {f42c5a73-1a26-4be2-950e-d607ac6c6f9f} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.853713 [ 5184 ] {50a834f6-c377-429a-b411-221e89a3666a} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.95 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.864794 [ 4535 ] {3f3f997b-01cd-40ce-8d76-04cc0c9aa898} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.5]:9000 -> [::ffff:127.0.0.1]:53946). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:17.866514 [ 4837 ] {08329455-d687-4a70-86a1-b1dc87f080e4} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.78 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.868454 [ 5186 ] {ab303059-a412-4925-b67e-d2099ee52191} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:34374). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:17.868545 [ 4535 ] {3f3f997b-01cd-40ce-8d76-04cc0c9aa898} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.79 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.867752 [ 4178 ] {254c59a5-60e9-4078-b8f1-1ee98d138211} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.16 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37996) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.869923 [ 4835 ] {8962c545-53e4-46a9-a1f6-513ed5790701} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.68 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34784) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.873253 [ 5186 ] {ab303059-a412-4925-b67e-d2099ee52191} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.78 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.873489 [ 3925 ] {8c531945-1368-4f3a-98fa-694099fa68e6} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.878713 [ 4832 ] {592a98a5-2565-461d-962e-b8197c4fe417} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.883627 [ 4838 ] {be620a69-658e-4120-ab8d-842db277c827} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:43934). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:17.884934 [ 4838 ] {be620a69-658e-4120-ab8d-842db277c827} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.57 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.883650 [ 5181 ] {9b56c16e-004d-4d4d-b186-8dc3e3ab4fa3} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.04 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38880) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.885275 [ 5179 ] {a89cb76a-f84e-4b97-8efb-498d466f64d4} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.9]:9000 -> [::ffff:127.0.0.1]:46080). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:17.884288 [ 3928 ] {1520e1e6-3672-4d76-a5a6-89b7d73c1e20} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.99 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53940) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.886541 [ 4837 ] {08329455-d687-4a70-86a1-b1dc87f080e4} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.887728 [ 4177 ] {13e8b5c5-2836-46fa-8f5d-5b0832d8b770} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.887767 [ 4534 ] {ff29cee3-3302-488d-a9dc-658c4646b9af} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.886651 [ 5179 ] {a89cb76a-f84e-4b97-8efb-498d466f64d4} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.02 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.889615 [ 5184 ] {50a834f6-c377-429a-b411-221e89a3666a} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.894365 [ 4835 ] {8962c545-53e4-46a9-a1f6-513ed5790701} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.68 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.892066 [ 4839 ] {fd87ee26-4092-445f-b24e-2a7f92ad43bb} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.04 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59916) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.895565 [ 5186 ] {ab303059-a412-4925-b67e-d2099ee52191} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.895616 [ 4535 ] {3f3f997b-01cd-40ce-8d76-04cc0c9aa898} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.897542 [ 4838 ] {be620a69-658e-4120-ab8d-842db277c827} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.895622 [ 5181 ] {9b56c16e-004d-4d4d-b186-8dc3e3ab4fa3} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.04 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.906218 [ 5179 ] {a89cb76a-f84e-4b97-8efb-498d466f64d4} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.907298 [ 4836 ] {5238363a-b2d7-418a-ad2a-077ee46b69a9} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.03 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) (version 23.8.16.41.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38870) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.907376 [ 4835 ] {8962c545-53e4-46a9-a1f6-513ed5790701} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.907648 [ 5181 ] {9b56c16e-004d-4d4d-b186-8dc3e3ab4fa3} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:17.909315 [ 4839 ] {fd87ee26-4092-445f-b24e-2a7f92ad43bb} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.04 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:17.914131 [ 4839 ] {fd87ee26-4092-445f-b24e-2a7f92ad43bb} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.568655 [ 4180 ] {2657b4bb-c61f-4a75-96b0-93bd938d1259} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.36 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.582129 [ 4182 ] {72586ce3-e4c1-4539-af43-db9ffd65f74d} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.35 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.595738 [ 4836 ] {5238363a-b2d7-418a-ad2a-077ee46b69a9} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.03 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.617884 [ 4180 ] {2657b4bb-c61f-4a75-96b0-93bd938d1259} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.625869 [ 4182 ] {72586ce3-e4c1-4539-af43-db9ffd65f74d} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.628958 [ 4836 ] {5238363a-b2d7-418a-ad2a-077ee46b69a9} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.631878 [ 4533 ] {fa811a6c-a9a5-44a6-88bc-119cc6d4b433} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:49996). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:19.633075 [ 4533 ] {fa811a6c-a9a5-44a6-88bc-119cc6d4b433} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.64 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.652618 [ 4533 ] {fa811a6c-a9a5-44a6-88bc-119cc6d4b433} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.707168 [ 4530 ] {fcf891c0-032d-45ca-b18f-9de058a4b71e} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.25 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.730800 [ 4530 ] {fcf891c0-032d-45ca-b18f-9de058a4b71e} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.732631 [ 4529 ] {3e2bb08b-c604-40f0-b972-b1a72180b7b5} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:49760). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:19.735147 [ 4529 ] {3e2bb08b-c604-40f0-b972-b1a72180b7b5} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.19 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.760540 [ 4529 ] {3e2bb08b-c604-40f0-b972-b1a72180b7b5} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.773345 [ 4179 ] {e2fa0c66-49dd-49b2-883c-92594962b9b1} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:49988). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:19.774946 [ 4179 ] {e2fa0c66-49dd-49b2-883c-92594962b9b1} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.96 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.807858 [ 4176 ] {69fca91e-4b0e-471e-84f6-34d2cfec7c30} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:49750). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:19.808158 [ 4181 ] {f5c52405-5dcb-43d8-b0b3-64b838755d6c} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.24 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.810344 [ 4178 ] {254c59a5-60e9-4078-b8f1-1ee98d138211} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.16 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.810433 [ 4176 ] {69fca91e-4b0e-471e-84f6-34d2cfec7c30} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.08 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.837847 [ 4179 ] {e2fa0c66-49dd-49b2-883c-92594962b9b1} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.866155 [ 3927 ] {e467c5e8-d6e3-4331-937c-86f6fd7949e0} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:51496). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:19.880747 [ 4181 ] {f5c52405-5dcb-43d8-b0b3-64b838755d6c} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.881276 [ 3927 ] {e467c5e8-d6e3-4331-937c-86f6fd7949e0} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.81 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.890170 [ 3923 ] {45cccf4f-87ba-4b1b-afc9-b2295e761997} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:49980). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:19.902004 [ 3923 ] {45cccf4f-87ba-4b1b-afc9-b2295e761997} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.93 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.907006 [ 4178 ] {254c59a5-60e9-4078-b8f1-1ee98d138211} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.907706 [ 4176 ] {69fca91e-4b0e-471e-84f6-34d2cfec7c30} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.908632 [ 4532 ] {fd65b6fb-21ee-4971-83e6-6c9eb36d8c60} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.32 GiB (attempt to allocate chunk of 1050624 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<unsigned long, StringHashMapCell<unsigned long, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030095bf1 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x00000000300957e7 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.909549 [ 3927 ] {e467c5e8-d6e3-4331-937c-86f6fd7949e0} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.916179 [ 3928 ] {1520e1e6-3672-4d76-a5a6-89b7d73c1e20} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.99 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:19.929119 [ 3923 ] {45cccf4f-87ba-4b1b-afc9-b2295e761997} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.932835 [ 4532 ] {fd65b6fb-21ee-4971-83e6-6c9eb36d8c60} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:19.952471 [ 3928 ] {1520e1e6-3672-4d76-a5a6-89b7d73c1e20} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:20.257993 [ 3057 ] {ca54a6b2-91ce-4c85-9c77-fffcc87a89b5} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:49734). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:20.258221 [ 3057 ] {ca54a6b2-91ce-4c85-9c77-fffcc87a89b5} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 28.33 GiB (attempt to allocate chunk of 0 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x000000002fd76cda in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a13 in /usr/bin/clickhouse
9. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
10. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
11. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
13. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
14. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
15. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
16. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
18. ? @ 0x00007f9fadd0cac3 in ?
19. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:20.281235 [ 3924 ] {3a79b3df-0f6c-4cc6-a5bd-a3c65cfd1ede} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:49738). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000001dd64b3b in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:96: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000001dd62b6a in /usr/bin/clickhouse
4. ./build_docker/./src/IO/WriteBuffer.h:60: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x0000000034f0956c in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: DB::TCPHandler::sendLogs() @ 0x0000000034ee8348 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecf158 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse
12. ? @ 0x00007f9fadd0cac3 in ?
13. ? @ 0x00007f9fadd9da04 in ?
 (version 23.8.16.41.altinitystable (altinity build))
2024.09.24 09:21:20.283603 [ 3057 ] {ca54a6b2-91ce-4c85-9c77-fffcc87a89b5} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:20.284318 [ 3924 ] {3a79b3df-0f6c-4cc6-a5bd-a3c65cfd1ede} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.99 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:20.316478 [ 3924 ] {3a79b3df-0f6c-4cc6-a5bd-a3c65cfd1ede} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:21:20.392699 [ 3926 ] {f4c557dd-7035-42b3-8c6b-3cc7ec168fcc} <Error> TCPHandler: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.67 GiB (attempt to allocate chunk of 1052672 bytes), maximum: 27.55 GiB. OvercommitTracker decision: Memory overcommit has freed not enough memory.: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*, char const*, String, long&, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001d60ab71 in /usr/bin/clickhouse
3. ./build_docker/./src/Common/MemoryTracker.cpp:340: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d608239 in /usr/bin/clickhouse
4. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
5. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
6. ./build_docker/./src/Common/MemoryTracker.cpp:0: MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000001d607bf4 in /usr/bin/clickhouse
7. ./build_docker/./src/Common/CurrentMemoryTracker.cpp:59: CurrentMemoryTracker::alloc(long) @ 0x000000001d534f9d in /usr/bin/clickhouse
8. ./build_docker/./src/Common/Allocator.h:102: Allocator<true, true>::alloc(unsigned long, unsigned long) @ 0x000000001d53172f in /usr/bin/clickhouse
9. ./build_docker/./src/Common/HashTable/HashTable.h:521: HashTable<StringKey24, StringHashMapCell<StringKey24, char*>, StringHashTableHash, StringHashTableGrower<8ul>, Allocator<true, true>>::alloc(StringHashTableGrower<8ul> const&) @ 0x0000000030096371 in /usr/bin/clickhouse
10. ./build_docker/./src/Common/HashTable/StringHashTable.h:228: StringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>>::StringHashTable() @ 0x0000000030095926 in /usr/bin/clickhouse
11. ./build_docker/./src/Common/HashTable/TwoLevelStringHashTable.h:43: TwoLevelStringHashTable<StringHashMapSubMaps<char*, Allocator<true, true>>, StringHashMap<char*, Allocator<true, true>>, 8ul>::TwoLevelStringHashTable<StringHashMap<char*, Allocator<true, true>>>(StringHashMap<char*, Allocator<true, true>> const&) @ 0x00000000300946b9 in /usr/bin/clickhouse
12. ./build_docker/./src/Common/HashTable/TwoLevelStringHashMap.h:0: DB::AggregatedDataVariants::convertToTwoLevel() @ 0x000000002fd7e03e in /usr/bin/clickhouse
13. ./build_docker/./src/Interpreters/Aggregator.cpp:0: DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x000000002fda2a04 in /usr/bin/clickhouse
14. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:669: DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000035741b19 in /usr/bin/clickhouse
15. ./build_docker/./src/Processors/Transforms/AggregatingTransform.cpp:0: DB::AggregatingTransform::work() @ 0x000000003573b4b1 in /usr/bin/clickhouse
16. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::ExecutionThreadContext::executeTask() @ 0x0000000035012b2e in /usr/bin/clickhouse
17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:251: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000034ff53dc in /usr/bin/clickhouse
18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:359: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000034ff8262 in /usr/bin/clickhouse
19. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000001d8a9b9c in /usr/bin/clickhouse
20. ./build_docker/./src/Common/ThreadPool.h:243: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x000000001d8b4000 in /usr/bin/clickhouse
21. ./build_docker/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000001d8a0692 in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000001d8ae96d in /usr/bin/clickhouse
23. ? @ 0x00007f9fadd0cac3 in ?
24. ? @ 0x00007f9fadd9da04 in ?

2024.09.24 09:21:20.420985 [ 3926 ] {f4c557dd-7035-42b3-8c6b-3cc7ec168fcc} <Warning> TCPHandler: Client has gone away.
2024.09.24 09:23:30.406327 [ 744 ] {} <Warning> Application: Closed all listening sockets. Waiting for 3 outstanding connections.
2024.09.24 09:23:35.311998 [ 744 ] {} <Warning> Application: Closed connections. But 3 remain. Tip: To increase wait time add to config: <shutdown_wait_unfinished>60</shutdown_wait_unfinished>
2024.09.24 09:23:35.312100 [ 744 ] {} <Warning> Application: Will shutdown forcefully.