2024.03.05 17:10:07.957758 [ 350 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.03.05 17:10:07.957975 [ 350 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.03.05 17:10:07.960046 [ 350 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.03.05 17:10:09.309435 [ 350 ] {} <Warning> KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper.
2024.03.05 17:10:09.310490 [ 350 ] {} <Warning> KeeperLogStore: Removing all changelogs
2024.03.05 17:10:09.311793 [ 350 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.03.05 17:10:09.316788 [ 350 ] {} <Warning> KeeperStateManager: No state was read
2024.03.05 17:10:09.316853 [ 350 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.03.05 17:10:09.316884 [ 350 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.03.05 17:10:09.368193 [ 592 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.03.05 17:10:09.418175 [ 350 ] {} <Warning> Access(local directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.03.05 17:10:09.418226 [ 350 ] {} <Warning> Access(local directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.03.05 17:10:09.500565 [ 350 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.03.05 17:10:33.384285 [ 595 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.03.05 17:10:33.395934 [ 591 ] {} <Warning> KeeperLogStore: Raft server is not set in LogStore.
2024.03.05 17:10:33.472046 [ 350 ] {} <Warning> KeeperLogStore: Changelog is shut down
2024.03.05 17:10:36.455859 [ 745 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.03.05 17:10:36.456072 [ 745 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.03.05 17:10:36.458101 [ 745 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.03.05 17:10:37.601936 [ 745 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.03.05 17:10:37.606803 [ 745 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.03.05 17:10:37.606846 [ 745 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.03.05 17:10:37.658508 [ 963 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.03.05 17:10:38.070899 [ 1043 ] {} <Error> Container already exists, returning the existing container: std::exception. Code: 1001, type: Azure::Storage::StorageException, e.what() = 409 The specified container already exists.
The specified container already exists.
RequestId:9fb433a0-95df-411c-9a5a-580ee22e76b6
Time:2024-03-05T15:10:37.803Z
Request ID: 9fb433a0-95df-411c-9a5a-580ee22e76b6, Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: std::runtime_error::runtime_error(String const&) @ 0x27276c3e in /usr/bin/clickhouse
1. ./build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/exception.hpp:29: Azure::Storage::StorageException::CreateFromResponse(std::unique_ptr<Azure::Core::Http::RawResponse, std::default_delete<Azure::Core::Http::RawResponse>>) @ 0x205540be in /usr/bin/clickhouse
2. ./build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/protocol/blob_rest_client.hpp:4760: Azure::Storage::Blobs::_detail::BlobRestClient::BlobContainer::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobRestClient::BlobContainer::CreateBlobContainerOptions const&, Azure::Core::Context const&) @ 0x204ce352 in /usr/bin/clickhouse
3. ./build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/nullable.hpp:91: Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const @ 0x204ca6d4 in /usr/bin/clickhouse
4. ./build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_service_client.cpp:0: Azure::Storage::Blobs::BlobServiceClient::CreateBlobContainer(String const&, Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const @ 0x204f67d7 in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::getAzureBlobContainerClient(Poco::Util::AbstractConfiguration const&, String const&) @ 0x1d08a1d8 in /usr/bin/clickhouse
6. ./build_docker/./src/Disks/ObjectStorages/AzureBlobStorage/registerDiskAzureBlobStorage.cpp:31: std::shared_ptr<DB::IDisk> std::__function::__policy_invoker<std::shared_ptr<DB::IDisk> (String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerDiskAzureBlobStorage(DB::DiskFactory&, bool)::$_0, std::shared_ptr<DB::IDisk> (String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&)>>(std::__function::__policy_storage const*, String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>&&, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&) @ 0x1d0871d8 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:0: DB::DiskFactory::create(String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&) const @ 0x1cfd9614 in /usr/bin/clickhouse
8. ./build_docker/./src/Disks/DiskSelector.cpp:49: DB::DiskSelector::initialize(Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>) @ 0x1d2a96a1 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Context::getDiskSelector(std::lock_guard<std::mutex>&) const @ 0x1d252b33 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/Context.cpp:3136: DB::Context::getStoragePolicySelector(std::lock_guard<std::mutex>&) const @ 0x1d234f01 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/Context.cpp:3068: DB::Context::getStoragePolicy(String const&) const @ 0x1d25308b in /usr/bin/clickhouse
12. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:0: DB::MergeTreeData::getStoragePolicy() const @ 0x1f3bcea8 in /usr/bin/clickhouse
13. ./build_docker/./src/Storages/MergeTree/MergeTreeData.h:912: DB::MergeTreeData::initializeDirectoriesAndFormatVersion(String const&, bool, String const&, bool) @ 0x1f3b728d in /usr/bin/clickhouse
14. ./build_docker/./src/Storages/StorageMergeTree.cpp:112: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, bool, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, bool) @ 0x1f7ac389 in /usr/bin/clickhouse
15. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:0: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, bool const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, bool const&, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, bool const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, bool const&) @ 0x1f7ab973 in /usr/bin/clickhouse
16. ./build_docker/./src/Storages/MergeTree/registerStorageMergeTree.cpp:0: DB::create(DB::StorageFactory::Arguments const&) @ 0x1f7a83b5 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:0: std::shared_ptr<DB::IStorage> std::__function::__policy_invoker<std::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<std::shared_ptr<DB::IStorage> (*)(DB::StorageFactory::Arguments const&), std::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::StorageFactory::Arguments const&) @ 0x1c57b703 in /usr/bin/clickhouse
18. ./build_docker/./src/Storages/StorageFactory.cpp:0: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x1ecb0f4a in /usr/bin/clickhouse
19. ./build_docker/./src/Databases/DatabaseOnDisk.cpp:0: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, bool) @ 0x1cf3f0a5 in /usr/bin/clickhouse
20. ./build_docker/./src/Databases/DatabaseOrdinary.cpp:0: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x1cf63bf1 in /usr/bin/clickhouse
21. ./build_docker/./src/Databases/TablesLoader.cpp:209: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::TablesLoader::startLoadingTables(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, std::shared_ptr<DB::Context>, std::vector<DB::StorageID, std::allocator<DB::StorageID>> const&, unsigned long)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x1cfcb52d in /usr/bin/clickhouse
22. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x1486be6a in /usr/bin/clickhouse
23. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x14870898 in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x14870802 in /usr/bin/clickhouse
25. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
26. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
27. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
28. ? @ 0x7f4c26c56609 in ?
29. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:10:38.563692 [ 745 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.03.05 17:16:18.393222 [ 752 ] {bbd89d38-e086-426e-b78c-26fa48da4392} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:56268) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 = (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1cd2febd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff6f9e9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1ff263d6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1ff49adc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e1f0e86 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5af2ce in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae884 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5b36eb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae8dc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c824 in /usr/bin/clickhouse
18. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e530e94 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e52c48d in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e12bf52 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e12830e in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e124046 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1ee33f in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1eb5e4 in /usr/bin/clickhouse
25. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e0d7cc9 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0d6cc4 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e60ea17 in /usr/bin/clickhouse
28. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e60c00c in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99d414 in /usr/bin/clickhouse
30. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse

2024.03.05 17:16:18.401974 [ 752 ] {bbd89d38-e086-426e-b78c-26fa48da4392} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1cd2febd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff6f9e9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1ff263d6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1ff49adc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e1f0e86 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5af2ce in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae884 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5b36eb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae8dc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c824 in /usr/bin/clickhouse
18. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e530e94 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e52c48d in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e12bf52 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e12830e in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e124046 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1ee33f in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1eb5e4 in /usr/bin/clickhouse
25. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e0d7cc9 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0d6cc4 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e60ea17 in /usr/bin/clickhouse
28. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e60c00c in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99d414 in /usr/bin/clickhouse
30. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse

2024.03.05 17:16:18.683500 [ 752 ] {e344283d-fa65-4b94-a361-ff00c74f0f50} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:56268) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1cd2febd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff6f9e9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1ff263d6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1ff49adc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e1f0e86 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e60f427 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e60c00c in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99d414 in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
17. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
18. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
19. ? @ 0x7f4c26c56609 in ?
20. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:16:18.684630 [ 752 ] {e344283d-fa65-4b94-a361-ff00c74f0f50} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1cd2febd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff6f9e9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1ff263d6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1ff49adc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e1f0e86 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e60f427 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e60c00c in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99d414 in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
17. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
18. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
19. ? @ 0x7f4c26c56609 in ?
20. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:16:18.979692 [ 752 ] {18f2ec51-f029-4428-8e4f-e56fda7bab86} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:56268) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) FROM test.hits WHERE NOT ignore(AdvEngineID);), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:16:19.002352 [ 752 ] {18f2ec51-f029-4428-8e4f-e56fda7bab86} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:16:19.375293 [ 752 ] {f3f952de-9a01-43de-a564-8094c0b2d24b} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:56268) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID > (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1cd2febd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff6f9e9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1ff263d6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1ff49adc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e1f0e86 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5af2ce in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae884 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5b36eb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae8dc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e530e94 in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e52c48d in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e12bf52 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e12830e in /usr/bin/clickhouse
21. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e124046 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1ee33f in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1eb5e4 in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e0d7cc9 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0d6cc4 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e60ea17 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e60c00c in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99d414 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse

2024.03.05 17:16:19.376355 [ 752 ] {f3f952de-9a01-43de-a564-8094c0b2d24b} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1cd2febd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff6f9e9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1ff263d6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1ff49adc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e1f0e86 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5af2ce in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae884 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5b36eb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5ae8dc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e53c6af in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e53c819 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e530e94 in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e52c48d in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e12bf52 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e12830e in /usr/bin/clickhouse
21. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e124046 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1ee33f in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e1eb5e4 in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e0d7cc9 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0d6cc4 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e60ea17 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e60c00c in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99d414 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse

2024.03.05 17:16:19.649890 [ 752 ] {93ebc973-f79c-4726-bc2f-5a5411bed4bb} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:56268) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID < (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:16:19.676628 [ 752 ] {93ebc973-f79c-4726-bc2f-5a5411bed4bb} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1cd30034 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fbe4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:47.960887 [ 2564 ] {dd4d956e-7c6a-4d7d-8974-0f615f944d62} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.00400837 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:36536) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:47.960894 [ 2563 ] {c43d03bf-0713-4391-889d-b12b26a347bb} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.012008438 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33370) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:48.091502 [ 2563 ] {c43d03bf-0713-4391-889d-b12b26a347bb} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.012008438 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:48.195867 [ 2564 ] {dd4d956e-7c6a-4d7d-8974-0f615f944d62} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.00400837 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:48.196174 [ 2561 ] {745a77ed-7568-400a-8ba9-ac6cf91ea5c4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.586326795 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:38682) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:48.390126 [ 2564 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:36536). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:48.390327 [ 2561 ] {745a77ed-7568-400a-8ba9-ac6cf91ea5c4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.586326795 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:48.390501 [ 2563 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:33370). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:49.845054 [ 2561 ] {cdd9cc51-fd75-426e-bcef-31e2d0b497ef} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.087000676 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:53208) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:49.929176 [ 2561 ] {cdd9cc51-fd75-426e-bcef-31e2d0b497ef} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.087000676 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:50.019897 [ 2564 ] {05952df2-eebf-4ec3-af52-ffec2e65a1ab} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.08400906 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45230) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:50.023967 [ 3147 ] {139f2837-1aef-497f-adf7-733135d91a6a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.158674311 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38328) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:50.026401 [ 2563 ] {43cbdd52-d476-4251-b243-37abe97f5ff9} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.076008993 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55386) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:50.098431 [ 2564 ] {05952df2-eebf-4ec3-af52-ffec2e65a1ab} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.08400906 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:50.099316 [ 2564 ] {05952df2-eebf-4ec3-af52-ffec2e65a1ab} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:50.099600 [ 3147 ] {139f2837-1aef-497f-adf7-733135d91a6a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.158674311 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:50.100740 [ 3147 ] {139f2837-1aef-497f-adf7-733135d91a6a} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:50.289976 [ 2563 ] {43cbdd52-d476-4251-b243-37abe97f5ff9} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:55386). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1df4362e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1df42c18 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f9b3f7c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f9a8548 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99ed50 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
14. ? @ 0x7f4c26c56609 in ?
15. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:50.290320 [ 2563 ] {43cbdd52-d476-4251-b243-37abe97f5ff9} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.076008993 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:50.290823 [ 2563 ] {43cbdd52-d476-4251-b243-37abe97f5ff9} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:51.397556 [ 2561 ] {56cd36c2-a232-47e0-9781-ef31d950673d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016008505 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38344) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.398593 [ 3147 ] {d6254be9-f1ae-4525-ac20-b2013582a98e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.060008873 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55396) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.399238 [ 3150 ] {1fc14914-5c52-4768-94ff-e3fea00f9cd0} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.032008639 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45244) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.399631 [ 3149 ] {60b82331-689b-4f13-9212-06627e5ab214} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.044008739 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44184) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.454129 [ 2561 ] {56cd36c2-a232-47e0-9781-ef31d950673d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016008505 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.455274 [ 3149 ] {60b82331-689b-4f13-9212-06627e5ab214} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.044008739 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.455860 [ 3147 ] {d6254be9-f1ae-4525-ac20-b2013582a98e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.060008873 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.460499 [ 3150 ] {1fc14914-5c52-4768-94ff-e3fea00f9cd0} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.032008639 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.463590 [ 2564 ] {524660ee-679b-49b0-8b16-63a723e023dc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.218728891 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:53220) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.468265 [ 3150 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:45244). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:51.470083 [ 2564 ] {524660ee-679b-49b0-8b16-63a723e023dc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.218728891 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:51.470091 [ 3147 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:55396). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:53.208062 [ 3150 ] {5dc6335d-289e-4080-98bd-5dfbb4da9705} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.124009424 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44188) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:53.210251 [ 2564 ] {7d0dac26-80df-446a-9cf7-9c5e8d862ec4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.160009727 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38350) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:53.211254 [ 3149 ] {126c8270-6b67-4395-bfeb-1dad4b314f8d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.148009626 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35438) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:53.211688 [ 2561 ] {db7d2b0c-121a-43e7-bb6b-277f4e0ffd5c} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.120009391 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55406) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:53.212993 [ 3147 ] {92816660-d2b2-4bf1-95bd-2cf99093d1ab} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100009223 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45258) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:54.351400 [ 2561 ] {db7d2b0c-121a-43e7-bb6b-277f4e0ffd5c} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.120009391 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:54.389543 [ 3147 ] {92816660-d2b2-4bf1-95bd-2cf99093d1ab} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100009223 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:54.391483 [ 3149 ] {126c8270-6b67-4395-bfeb-1dad4b314f8d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.148009626 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:54.413173 [ 3150 ] {5dc6335d-289e-4080-98bd-5dfbb4da9705} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.124009424 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:54.430734 [ 2564 ] {7d0dac26-80df-446a-9cf7-9c5e8d862ec4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.160009727 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:54.468926 [ 2563 ] {3c5baffa-8dcd-45b5-beef-d56ba2763345} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.630786211 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:53230) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:54.535102 [ 2564 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:38350). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:54.537398 [ 2561 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:55406). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:54.539224 [ 2563 ] {3c5baffa-8dcd-45b5-beef-d56ba2763345} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.630786211 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.091155 [ 2563 ] {31ea6a94-cbb3-4ee5-b923-2575e0208f8a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.184133827 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:53240) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.220318 [ 2563 ] {31ea6a94-cbb3-4ee5-b923-2575e0208f8a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.184133827 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.411173 [ 3147 ] {17e2fdfe-a5ce-4599-9f07-90d8d535b7a4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.000008413 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35444) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.415373 [ 3150 ] {cdfe2d61-b81d-45ea-94a8-5dff0dde1733} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.112009356 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44202) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.441557 [ 3153 ] {7ee73c4d-e270-4f53-916d-8fc4d9a8855d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.228010333 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43094) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.446814 [ 2564 ] {0f732c45-fa73-4819-8532-a03ba2178058} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.072009019 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55422) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.446932 [ 3149 ] {71bfae0a-4e26-4a8f-88aa-eeb4ebab4b61} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.004008447 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38362) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.463340 [ 2561 ] {71c59d91-16c7-45c7-b076-4eba265cf143} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.028008649 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45274) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.606051 [ 3147 ] {17e2fdfe-a5ce-4599-9f07-90d8d535b7a4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.000008413 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.608509 [ 3153 ] {7ee73c4d-e270-4f53-916d-8fc4d9a8855d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.228010333 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.615003 [ 3153 ] {7ee73c4d-e270-4f53-916d-8fc4d9a8855d} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:56.615229 [ 3147 ] {17e2fdfe-a5ce-4599-9f07-90d8d535b7a4} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:56.617651 [ 2564 ] {0f732c45-fa73-4819-8532-a03ba2178058} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:55422). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1df4362e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1df42c18 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f9b3f7c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f9a8548 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99ed50 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
14. ? @ 0x7f4c26c56609 in ?
15. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:56.618615 [ 2564 ] {0f732c45-fa73-4819-8532-a03ba2178058} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.072009019 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.620582 [ 3150 ] {cdfe2d61-b81d-45ea-94a8-5dff0dde1733} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.112009356 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.626716 [ 2564 ] {0f732c45-fa73-4819-8532-a03ba2178058} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:56.629185 [ 3150 ] {cdfe2d61-b81d-45ea-94a8-5dff0dde1733} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:56.634753 [ 3149 ] {71bfae0a-4e26-4a8f-88aa-eeb4ebab4b61} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.004008447 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.640729 [ 3149 ] {71bfae0a-4e26-4a8f-88aa-eeb4ebab4b61} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:56.641359 [ 2561 ] {71c59d91-16c7-45c7-b076-4eba265cf143} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.028008649 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:56.642655 [ 2561 ] {71c59d91-16c7-45c7-b076-4eba265cf143} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:57.912618 [ 2563 ] {fa19f7be-e85e-449f-afbf-7974b3184426} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.180046013 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:53250) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.024088 [ 2563 ] {fa19f7be-e85e-449f-afbf-7974b3184426} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.180046013 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.250714 [ 3157 ] {2b591f6c-91d0-4eab-9881-dcaa3e2ad3fa} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.036008731 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45288) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.251275 [ 3160 ] {f6540bd5-4b4e-4347-9002-7fa83e20b8e9} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.13200954 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43098) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.251673 [ 3155 ] {0372d01d-4498-40eb-83f4-d8846b646b2f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.188010012 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55428) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.259092 [ 3159 ] {8b14941f-0d50-4770-aa57-3b5d37244a7a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.128009507 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44214) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.259766 [ 3158 ] {89358f45-cc9a-4f17-8e48-d1ae98ce16d5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.212010215 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35452) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.265534 [ 3161 ] {359af7c4-a236-471d-9ffe-45682a008339} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.088009169 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37950) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.270629 [ 3156 ] {4387a130-cfe1-41f9-a331-9a1c547b2ccb} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.160009776 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38378) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.448767 [ 3160 ] {f6540bd5-4b4e-4347-9002-7fa83e20b8e9} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.13200954 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.461763 [ 3160 ] {f6540bd5-4b4e-4347-9002-7fa83e20b8e9} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:58.487715 [ 3161 ] {359af7c4-a236-471d-9ffe-45682a008339} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.088009169 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.491006 [ 3157 ] {2b591f6c-91d0-4eab-9881-dcaa3e2ad3fa} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.036008731 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.493311 [ 3161 ] {359af7c4-a236-471d-9ffe-45682a008339} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:58.494929 [ 3158 ] {89358f45-cc9a-4f17-8e48-d1ae98ce16d5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.212010215 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.505448 [ 3157 ] {2b591f6c-91d0-4eab-9881-dcaa3e2ad3fa} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:58.515952 [ 3158 ] {89358f45-cc9a-4f17-8e48-d1ae98ce16d5} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:58.517388 [ 3159 ] {8b14941f-0d50-4770-aa57-3b5d37244a7a} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:44214). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1df4362e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1df42c18 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f9b3f7c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f9a8548 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99ed50 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
14. ? @ 0x7f4c26c56609 in ?
15. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:58.518671 [ 3159 ] {8b14941f-0d50-4770-aa57-3b5d37244a7a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.128009507 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.523806 [ 3159 ] {8b14941f-0d50-4770-aa57-3b5d37244a7a} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:58.525712 [ 3155 ] {0372d01d-4498-40eb-83f4-d8846b646b2f} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:55428). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1df4362e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1df42c18 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f9b3f7c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f9a8548 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99ed50 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
14. ? @ 0x7f4c26c56609 in ?
15. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:25:58.526200 [ 3155 ] {0372d01d-4498-40eb-83f4-d8846b646b2f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.188010012 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.527196 [ 3156 ] {4387a130-cfe1-41f9-a331-9a1c547b2ccb} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.160009776 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:25:58.551566 [ 3156 ] {4387a130-cfe1-41f9-a331-9a1c547b2ccb} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:25:58.557995 [ 3155 ] {0372d01d-4498-40eb-83f4-d8846b646b2f} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:00.141343 [ 3164 ] {f88a0638-9f99-4d9e-aae5-f8cc1de86c89} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.128009527 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:57616) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.141368 [ 2564 ] {32778dc9-e21e-40b7-b2ee-f705b9b6bf14} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.200010133 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:38384) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.144583 [ 3147 ] {e01c255f-9b83-47df-bb70-7d8d21cb924f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.172009896 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44218) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.153069 [ 3153 ] {00e40035-1577-4ed2-8c76-338b42fef91f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.220010301 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43104) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.162551 [ 3163 ] {f0624bc5-2c6f-483f-b695-1d2423cf49cd} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.056008919 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37964) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.167767 [ 3149 ] {62ba61e1-8fe2-48a4-b1c7-ef6133c24337} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208010199 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:45296) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.167803 [ 2561 ] {a29f184d-c21c-4044-b2e7-c5ee67eb8c4f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.256010605 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55442) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.180062 [ 3150 ] {ee1e44d2-85d9-4ee7-b6a5-e000384d6b83} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.136009593 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35456) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.291675 [ 3164 ] {f88a0638-9f99-4d9e-aae5-f8cc1de86c89} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.128009527 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.318217 [ 3163 ] {f0624bc5-2c6f-483f-b695-1d2423cf49cd} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.056008919 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.330624 [ 2564 ] {32778dc9-e21e-40b7-b2ee-f705b9b6bf14} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.200010133 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.334096 [ 2561 ] {a29f184d-c21c-4044-b2e7-c5ee67eb8c4f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.256010605 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.340544 [ 3150 ] {ee1e44d2-85d9-4ee7-b6a5-e000384d6b83} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.136009593 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.341902 [ 3153 ] {00e40035-1577-4ed2-8c76-338b42fef91f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.220010301 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.345754 [ 3147 ] {e01c255f-9b83-47df-bb70-7d8d21cb924f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.172009896 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.350057 [ 3149 ] {62ba61e1-8fe2-48a4-b1c7-ef6133c24337} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208010199 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:00.648506 [ 2563 ] {d464fd57-d060-4827-a12f-44e10df02b73} <Error> executeQuery: Code: 159. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Timeout exceeded: elapsed 1.256010605 seconds, maximum: 1: While executing MergeTreeThread. Stack trace:

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?
: While executing Remote. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:53258) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::readException(DB::ReadBuffer&, String const&, bool) @ 0x147cc2a7 in /usr/bin/clickhouse
3. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::Connection::receiveException() const @ 0x1f86bc09 in /usr/bin/clickhouse
4. ./build_docker/./src/Client/Connection.cpp:0: DB::Connection::receivePacket() @ 0x1f87478e in /usr/bin/clickhouse
5. ./build_docker/./src/Client/PacketReceiver.h:124: DB::PacketReceiver::Routine::operator()(boost::context::fiber&&) @ 0x1f89918f in /usr/bin/clickhouse
6. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: boost::context::detail::fiber_capture_record<boost::context::fiber, FiberStack&, DB::PacketReceiver::Routine>::run() @ 0x1f898c47 in /usr/bin/clickhouse
7. ./build_docker/./contrib/boost/boost/context/fiber_ucontext.hpp:74: void boost::context::detail::fiber_entry_func<boost::context::detail::fiber_capture_record<boost::context::fiber, FiberStack&, DB::PacketReceiver::Routine>>(void*) @ 0x1f89625a in /usr/bin/clickhouse

2024.03.05 17:26:00.682035 [ 3149 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:45296). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:26:00.688022 [ 2564 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:38384). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1df41e9d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1df41470 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f99c863 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
12. ? @ 0x7f4c26c56609 in ?
13. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:26:00.689388 [ 2563 ] {d464fd57-d060-4827-a12f-44e10df02b73} <Error> TCPHandler: Code: 159. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Timeout exceeded: elapsed 1.256010605 seconds, maximum: 1: While executing MergeTreeThread. Stack trace:

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?
: While executing Remote. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::readException(DB::ReadBuffer&, String const&, bool) @ 0x147cc2a7 in /usr/bin/clickhouse
3. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::Connection::receiveException() const @ 0x1f86bc09 in /usr/bin/clickhouse
4. ./build_docker/./src/Client/Connection.cpp:0: DB::Connection::receivePacket() @ 0x1f87478e in /usr/bin/clickhouse
5. ./build_docker/./src/Client/PacketReceiver.h:124: DB::PacketReceiver::Routine::operator()(boost::context::fiber&&) @ 0x1f89918f in /usr/bin/clickhouse
6. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:0: boost::context::detail::fiber_capture_record<boost::context::fiber, FiberStack&, DB::PacketReceiver::Routine>::run() @ 0x1f898c47 in /usr/bin/clickhouse
7. ./build_docker/./contrib/boost/boost/context/fiber_ucontext.hpp:74: void boost::context::detail::fiber_entry_func<boost::context::detail::fiber_capture_record<boost::context::fiber, FiberStack&, DB::PacketReceiver::Routine>>(void*) @ 0x1f89625a in /usr/bin/clickhouse

2024.03.05 17:26:02.368684 [ 3147 ] {9a5d94f8-dcdf-4a15-a8ba-f8f4941b816b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.331149754 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:37508) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.560144 [ 3147 ] {9a5d94f8-dcdf-4a15-a8ba-f8f4941b816b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.331149754 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.668472 [ 3150 ] {d2e69e32-b8a9-4a5d-8417-f62e64e9021f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.044008838 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48754) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.698378 [ 3157 ] {6171e9c0-2373-4b1c-a2c0-2479936eb2f0} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.43601216 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52394) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.700892 [ 3155 ] {98d3835c-e1bd-47f9-b60e-ef1f4725ee47} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.448012261 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52266) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.701419 [ 3156 ] {6a648b3f-83f2-453d-b733-4915b37608e2} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.14800972 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55918) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.705853 [ 2563 ] {09628f63-d2f7-4ac0-b219-433944c3f887} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.428012092 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33250) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.711807 [ 2564 ] {c3deb977-bb64-45ee-b304-b2037ad02fee} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.060008975 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48876) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.712865 [ 3153 ] {c3a5aa7c-74d0-4c18-a689-31ab68f3cb9e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208010227 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47060) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.720067 [ 3149 ] {417e6b0e-42c8-4d10-a27a-1da48be3a427} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.248010567 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49064) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.725703 [ 2561 ] {aee73832-3ec4-44e8-8ff4-7099c33e5208} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.176009957 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43546) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.912778 [ 3149 ] {417e6b0e-42c8-4d10-a27a-1da48be3a427} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.248010567 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.912868 [ 3150 ] {d2e69e32-b8a9-4a5d-8417-f62e64e9021f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.044008838 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.914109 [ 3155 ] {98d3835c-e1bd-47f9-b60e-ef1f4725ee47} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.448012261 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.914402 [ 3149 ] {417e6b0e-42c8-4d10-a27a-1da48be3a427} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.914741 [ 3150 ] {d2e69e32-b8a9-4a5d-8417-f62e64e9021f} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.914777 [ 3157 ] {6171e9c0-2373-4b1c-a2c0-2479936eb2f0} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:52394). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1df4362e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1df42c18 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f9b3f7c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f9a8548 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99ed50 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
14. ? @ 0x7f4c26c56609 in ?
15. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:26:02.915490 [ 3155 ] {98d3835c-e1bd-47f9-b60e-ef1f4725ee47} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.915751 [ 3157 ] {6171e9c0-2373-4b1c-a2c0-2479936eb2f0} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.43601216 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.916683 [ 2564 ] {c3deb977-bb64-45ee-b304-b2037ad02fee} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.060008975 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.917849 [ 3157 ] {6171e9c0-2373-4b1c-a2c0-2479936eb2f0} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.918017 [ 2564 ] {c3deb977-bb64-45ee-b304-b2037ad02fee} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.919057 [ 2563 ] {09628f63-d2f7-4ac0-b219-433944c3f887} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.428012092 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.920511 [ 2563 ] {09628f63-d2f7-4ac0-b219-433944c3f887} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.920848 [ 3156 ] {6a648b3f-83f2-453d-b733-4915b37608e2} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.14800972 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.921061 [ 3153 ] {c3a5aa7c-74d0-4c18-a689-31ab68f3cb9e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.208010227 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.921598 [ 3156 ] {6a648b3f-83f2-453d-b733-4915b37608e2} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.921730 [ 3153 ] {c3a5aa7c-74d0-4c18-a689-31ab68f3cb9e} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:02.922035 [ 2561 ] {aee73832-3ec4-44e8-8ff4-7099c33e5208} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.176009957 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd159eb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:02.923459 [ 2561 ] {aee73832-3ec4-44e8-8ff4-7099c33e5208} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:03.612029 [ 2542 ] {6b19b0e5-497f-44f6-9ba4-1285fcf16d3a} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 576.15 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:38666) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1cd30214 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fd0f in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.046335 [ 3147 ] {3419b4f1-fa89-41a7-9ee9-75ef65431371} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.045298738 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:37512) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.193958 [ 3147 ] {3419b4f1-fa89-41a7-9ee9-75ef65431371} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.045298738 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.220955 [ 2542 ] {6b19b0e5-497f-44f6-9ba4-1285fcf16d3a} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 576.15 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1cd30214 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fd0f in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.377310 [ 2563 ] {6c055059-198d-4bad-a58a-21aa9054b7be} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.433853357 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48770) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.455726 [ 3150 ] {938236ab-edc4-46fa-a867-725a30226d89} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.515545815 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52410) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.458245 [ 3149 ] {4d1b1700-216e-4cf0-adde-0055e9995606} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.514670975 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:47068) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.497853 [ 3153 ] {f03c9298-f1fc-4709-8b05-e8a9289cce83} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.550188142 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55928) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.497911 [ 3155 ] {150a3d40-00a5-4a83-a041-db78685fa432} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.547208556 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43558) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.510498 [ 3158 ] {7e4d1540-3d10-4ec7-9e25-5555cec8f406} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.551182003 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:33256) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.515554 [ 3157 ] {e7a04455-c607-4261-bcfa-1d1f0e3d280a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.560170517 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:48882) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.516107 [ 3156 ] {f24efad3-0573-4b73-bbcc-a977f45ac1ad} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.561583861 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:49074) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.516818 [ 2564 ] {6dbdd24d-0eca-425b-aa20-8db7d275e0d7} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.567461784 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37468) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.537408 [ 3159 ] {9ae4082a-a844-4168-bdc7-edc6cdd49321} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.577607939 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.33.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:52278) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.646270 [ 2563 ] {6c055059-198d-4bad-a58a-21aa9054b7be} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.433853357 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.658306 [ 3149 ] {4d1b1700-216e-4cf0-adde-0055e9995606} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.514670975 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.660895 [ 2563 ] {6c055059-198d-4bad-a58a-21aa9054b7be} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.667347 [ 3149 ] {4d1b1700-216e-4cf0-adde-0055e9995606} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.675492 [ 3153 ] {f03c9298-f1fc-4709-8b05-e8a9289cce83} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.550188142 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.682946 [ 3150 ] {938236ab-edc4-46fa-a867-725a30226d89} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:52410). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1df4362e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1df42c18 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f9b3f7c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f9a8548 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f99ed50 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f9b4368 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x23751da3 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x23752613 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x239adb3a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x239abef0 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x239aa508 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
14. ? @ 0x7f4c26c56609 in ?
15. clone @ 0x7f4c26b7b353 in ?
 (version 23.3.19.33.altinitystable (altinity build))
2024.03.05 17:26:04.683489 [ 3150 ] {938236ab-edc4-46fa-a867-725a30226d89} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.515545815 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.683917 [ 3153 ] {f03c9298-f1fc-4709-8b05-e8a9289cce83} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.687356 [ 3157 ] {e7a04455-c607-4261-bcfa-1d1f0e3d280a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.560170517 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.689425 [ 3156 ] {f24efad3-0573-4b73-bbcc-a977f45ac1ad} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.561583861 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.692739 [ 3150 ] {938236ab-edc4-46fa-a867-725a30226d89} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.706441 [ 2564 ] {6dbdd24d-0eca-425b-aa20-8db7d275e0d7} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.567461784 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.708537 [ 2564 ] {6dbdd24d-0eca-425b-aa20-8db7d275e0d7} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.709830 [ 3157 ] {e7a04455-c607-4261-bcfa-1d1f0e3d280a} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.710905 [ 3156 ] {f24efad3-0573-4b73-bbcc-a977f45ac1ad} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.715237 [ 3155 ] {150a3d40-00a5-4a83-a041-db78685fa432} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.547208556 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.718093 [ 3158 ] {7e4d1540-3d10-4ec7-9e25-5555cec8f406} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.551182003 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.726819 [ 3155 ] {150a3d40-00a5-4a83-a041-db78685fa432} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.729579 [ 3158 ] {7e4d1540-3d10-4ec7-9e25-5555cec8f406} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:04.733272 [ 3159 ] {9ae4082a-a844-4168-bdc7-edc6cdd49321} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.577607939 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cd17025 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cd16b51 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e2a341d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1fa169e3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1fa16649 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa2b236 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:04.735870 [ 3159 ] {9ae4082a-a844-4168-bdc7-edc6cdd49321} <Warning> TCPHandler: Client has gone away.
2024.03.05 17:26:13.323301 [ 2542 ] {645d493a-ec21-44fc-b371-4c270bc7c3b5} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 574.12 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.3.19.33.altinitystable (altinity build)) (from [::1]:38666) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1cd30214 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fd0f in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?

2024.03.05 17:26:13.483376 [ 2542 ] {645d493a-ec21-44fc-b371-4c270bc7c3b5} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 574.12 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x23912b3e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x1474e758 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1cd30214 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1cd2fd0f in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cd15cdc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1fa268c2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1fa17561 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1fa18e0e in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x14868268 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x1486dd32 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc66f4a7 in /usr/bin/clickhouse
11. ? @ 0x7f4c26c56609 in ?
12. clone @ 0x7f4c26b7b353 in ?