2024.03.20 05:54:54.223139 [ 350 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.03.20 05:54:54.223343 [ 350 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.03.20 05:54:54.225387 [ 350 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.03.20 05:54:55.458323 [ 350 ] {} <Warning> KeeperLogStore: No logs exists in /var/lib/clickhouse/coordination/logs. It's Ok if it's the first run of clickhouse-keeper.
2024.03.20 05:54:55.459571 [ 350 ] {} <Warning> KeeperLogStore: Removing all changelogs
2024.03.20 05:54:55.460874 [ 350 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.03.20 05:54:55.460914 [ 350 ] {} <Fatal> KeeperServer: Can't start NON-SECURE keeper server in FIPS mode, please check the config.
2024.03.20 05:54:55.465349 [ 350 ] {} <Warning> KeeperStateManager: No state was read
2024.03.20 05:54:55.465409 [ 350 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.03.20 05:54:55.465442 [ 350 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.03.20 05:54:55.516697 [ 591 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.03.20 05:54:55.559357 [ 350 ] {} <Warning> Access(local directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.03.20 05:54:55.559408 [ 350 ] {} <Warning> Access(local directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.03.20 05:54:55.641726 [ 350 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.03.20 05:55:20.036868 [ 594 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 125, Operation canceled
2024.03.20 05:55:20.047901 [ 590 ] {} <Warning> KeeperLogStore: Raft server is not set in LogStore.
2024.03.20 05:55:20.084078 [ 350 ] {} <Warning> KeeperLogStore: Changelog is shut down
2024.03.20 05:55:23.559073 [ 748 ] {} <Warning> Context: Server was built with sanitizer. It will work slowly.
2024.03.20 05:55:23.559272 [ 748 ] {} <Warning> Context: Server logging level is set to 'test' and performance is degraded. This cannot be used in production.
2024.03.20 05:55:23.561380 [ 748 ] {} <Warning> Context: The setting 'allow_remote_fs_zero_copy_replication' is enabled for MergeTree tables. But the feature of 'zero-copy replication' is under development and is not ready for production. The usage of this feature can lead to data corruption and loss. The setting should be disabled in production.
2024.03.20 05:55:24.829328 [ 748 ] {} <Warning> KeeperServer: Got 1000000000000000 value for setting 'reserved_log_items' which is bigger than int32_t max value, lowering value to 2147483647.
2024.03.20 05:55:24.829411 [ 748 ] {} <Fatal> KeeperServer: Can't start NON-SECURE keeper server in FIPS mode, please check the config.
2024.03.20 05:55:24.834454 [ 748 ] {} <Warning> RaftInstance: invalid election timeout lower bound detected, adjusted to 0
2024.03.20 05:55:24.834500 [ 748 ] {} <Warning> RaftInstance: invalid election timeout upper bound detected, adjusted to 0
2024.03.20 05:55:24.886059 [ 967 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2024.03.20 05:55:25.293530 [ 1048 ] {} <Error> Container already exists, returning the existing container: std::exception. Code: 1001, type: Azure::Storage::StorageException, e.what() = 409 The specified container already exists.
The specified container already exists.
RequestId:16299ee6-cc36-4890-8262-eda66c3faea1
Time:2024-03-19T16:55:25.020Z
Request ID: 16299ee6-cc36-4890-8262-eda66c3faea1, Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: std::runtime_error::runtime_error(String const&) @ 0x271b3c3e in /usr/bin/clickhouse
1. ./build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/exception.hpp:29: Azure::Storage::StorageException::CreateFromResponse(std::unique_ptr<Azure::Core::Http::RawResponse, std::default_delete<Azure::Core::Http::RawResponse>>) @ 0x204e6b3e in /usr/bin/clickhouse
2. ./build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/protocol/blob_rest_client.hpp:4760: Azure::Storage::Blobs::_detail::BlobRestClient::BlobContainer::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobRestClient::BlobContainer::CreateBlobContainerOptions const&, Azure::Core::Context const&) @ 0x20460dd2 in /usr/bin/clickhouse
3. ./build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/nullable.hpp:91: Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const @ 0x2045d154 in /usr/bin/clickhouse
4. ./build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_service_client.cpp:0: Azure::Storage::Blobs::BlobServiceClient::CreateBlobContainer(String const&, Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const @ 0x20489257 in /usr/bin/clickhouse
5. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: DB::getAzureBlobContainerClient(Poco::Util::AbstractConfiguration const&, String const&) @ 0x1d01c8b8 in /usr/bin/clickhouse
6. ./build_docker/./src/Disks/ObjectStorages/AzureBlobStorage/registerDiskAzureBlobStorage.cpp:31: std::shared_ptr<DB::IDisk> std::__function::__policy_invoker<std::shared_ptr<DB::IDisk> (String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerDiskAzureBlobStorage(DB::DiskFactory&, bool)::$_0, std::shared_ptr<DB::IDisk> (String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&)>>(std::__function::__policy_storage const*, String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>&&, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&) @ 0x1d0198b8 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:0: DB::DiskFactory::create(String const&, Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>, std::map<String, std::shared_ptr<DB::IDisk>, std::less<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::IDisk>>>> const&) const @ 0x1cf6bcf4 in /usr/bin/clickhouse
8. ./build_docker/./src/Disks/DiskSelector.cpp:49: DB::DiskSelector::initialize(Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr<DB::Context const>) @ 0x1d23bd81 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Context::getDiskSelector(std::lock_guard<std::mutex>&) const @ 0x1d1e5213 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/Context.cpp:3136: DB::Context::getStoragePolicySelector(std::lock_guard<std::mutex>&) const @ 0x1d1c75e1 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/Context.cpp:3068: DB::Context::getStoragePolicy(String const&) const @ 0x1d1e576b in /usr/bin/clickhouse
12. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:0: DB::MergeTreeData::getStoragePolicy() const @ 0x1f34f588 in /usr/bin/clickhouse
13. ./build_docker/./src/Storages/MergeTree/MergeTreeData.h:912: DB::MergeTreeData::initializeDirectoriesAndFormatVersion(String const&, bool, String const&, bool) @ 0x1f34996d in /usr/bin/clickhouse
14. ./build_docker/./src/Storages/StorageMergeTree.cpp:112: DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, bool, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, bool) @ 0x1f73ea69 in /usr/bin/clickhouse
15. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:0: std::shared_ptr<DB::StorageMergeTree> std::allocate_shared[abi:v15000]<DB::StorageMergeTree, std::allocator<DB::StorageMergeTree>, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, bool const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, bool const&, void>(std::allocator<DB::StorageMergeTree> const&, DB::StorageID const&, String const&, DB::StorageInMemoryMetadata&, bool const&, std::shared_ptr<DB::Context>&, String&, DB::MergeTreeData::MergingParams&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>&&, bool const&) @ 0x1f73e053 in /usr/bin/clickhouse
16. ./build_docker/./src/Storages/MergeTree/registerStorageMergeTree.cpp:0: DB::create(DB::StorageFactory::Arguments const&) @ 0x1f73aa95 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:0: std::shared_ptr<DB::IStorage> std::__function::__policy_invoker<std::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<std::shared_ptr<DB::IStorage> (*)(DB::StorageFactory::Arguments const&), std::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::StorageFactory::Arguments const&) @ 0x1c50dde3 in /usr/bin/clickhouse
18. ./build_docker/./src/Storages/StorageFactory.cpp:0: DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x1ec4362a in /usr/bin/clickhouse
19. ./build_docker/./src/Databases/DatabaseOnDisk.cpp:0: DB::createTableFromAST(DB::ASTCreateQuery, String const&, String const&, std::shared_ptr<DB::Context>, bool) @ 0x1ced1785 in /usr/bin/clickhouse
20. ./build_docker/./src/Databases/DatabaseOrdinary.cpp:0: DB::DatabaseOrdinary::loadTableFromMetadata(std::shared_ptr<DB::Context>, String const&, DB::QualifiedTableName const&, std::shared_ptr<DB::IAST> const&, DB::LoadingStrictnessLevel) @ 0x1cef62d1 in /usr/bin/clickhouse
21. ./build_docker/./src/Databases/TablesLoader.cpp:209: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::TablesLoader::startLoadingTables(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>&, std::shared_ptr<DB::Context>, std::vector<DB::StorageID, std::allocator<DB::StorageID>> const&, unsigned long)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x1cf5dc0d in /usr/bin/clickhouse
22. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x147fe4ea in /usr/bin/clickhouse
23. ./build_docker/./src/Common/ThreadPool.cpp:0: ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0x14802f18 in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:717: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x14802e82 in /usr/bin/clickhouse
25. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
26. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
27. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
28. ? @ 0x7fe874aa3609 in ?
29. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 05:55:25.729439 [ 748 ] {} <Warning> Context: Table system.session_log is enabled. It's unreliable and may contain garbage. Do not use it for any kind of security monitoring.
2024.03.20 06:01:20.753541 [ 756 ] {a9330607-23d6-4f18-bd9c-2bdceaf5d5d7} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:54222) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 = (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1ccc259d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff020c9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1feb8ab6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1fedc1bc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e183566 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5419ae in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540f64 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e545dcb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540fbc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4cef04 in /usr/bin/clickhouse
18. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e4c3574 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e4beb6d in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e0be632 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e0ba9ee in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e0b6726 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e180a1f in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e17dcc4 in /usr/bin/clickhouse
25. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e06a3a9 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0693a4 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e5a10f7 in /usr/bin/clickhouse
28. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e59e6ec in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f92faf4 in /usr/bin/clickhouse
30. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse

2024.03.20 06:01:20.754781 [ 756 ] {a9330607-23d6-4f18-bd9c-2bdceaf5d5d7} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3: While processing 1 = ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery3). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1ccc259d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff020c9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1feb8ab6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1fedc1bc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e183566 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5419ae in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540f64 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e545dcb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540fbc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
17. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:0: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4cef04 in /usr/bin/clickhouse
18. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e4c3574 in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e4beb6d in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e0be632 in /usr/bin/clickhouse
21. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e0ba9ee in /usr/bin/clickhouse
22. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e0b6726 in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e180a1f in /usr/bin/clickhouse
24. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e17dcc4 in /usr/bin/clickhouse
25. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e06a3a9 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0693a4 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e5a10f7 in /usr/bin/clickhouse
28. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e59e6ec in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f92faf4 in /usr/bin/clickhouse
30. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
31. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse

2024.03.20 06:01:21.059659 [ 756 ] {44dcb445-14bb-46a5-89b0-1ed98936d5ea} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:54222) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT 1 IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1ccc259d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff020c9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1feb8ab6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1fedc1bc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e183566 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e5a1b07 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e59e6ec in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f92faf4 in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
17. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
18. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
19. ? @ 0x7fe874aa3609 in ?
20. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:01:21.060896 [ 756 ] {44dcb445-14bb-46a5-89b0-1ed98936d5ea} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1ccc259d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff020c9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1feb8ab6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1fedc1bc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e183566 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e5a1b07 in /usr/bin/clickhouse
10. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e59e6ec in /usr/bin/clickhouse
11. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f92faf4 in /usr/bin/clickhouse
12. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
13. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
14. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
15. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
16. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
17. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
18. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
19. ? @ 0x7fe874aa3609 in ?
20. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:01:21.381279 [ 756 ] {a23d6c2a-2f01-44be-8c05-22ecd76577da} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:54222) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() IN (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) FROM test.hits WHERE NOT ignore(AdvEngineID);), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:01:21.404041 [ 756 ] {a23d6c2a-2f01-44be-8c05-22ecd76577da} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 15.00 million, current rows: 17.76 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:01:21.804575 [ 756 ] {891302b0-9c56-4baa-97d7-9075921e283e} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:54222) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID > (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1ccc259d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff020c9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1feb8ab6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1fedc1bc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e183566 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5419ae in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540f64 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e545dcb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540fbc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e4c3574 in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e4beb6d in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e0be632 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e0ba9ee in /usr/bin/clickhouse
21. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e0b6726 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e180a1f in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e17dcc4 in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e06a3a9 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0693a4 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e5a10f7 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e59e6ec in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f92faf4 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse

2024.03.20 06:01:21.806335 [ 756 ] {891302b0-9c56-4baa-97d7-9075921e283e} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows (controlled by 'max_rows_to_read' setting) exceeded, max rows: 1.00 million, current rows: 8.88 million: While processing (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6: While processing CounterID > ((SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID)) AS _subquery6). (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int) const @ 0x1ccc259d in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:0: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1ff020c9 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x1feb8ab6 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:0: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1fedc1bc in /usr/bin/clickhouse
8. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379: DB::InterpreterSelectWithUnionQuery::execute() @ 0x1e183566 in /usr/bin/clickhouse
9. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:190: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTSubquery const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e5419ae in /usr/bin/clickhouse
10. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:815: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540f64 in /usr/bin/clickhouse
11. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
12. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
13. ./build_docker/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: DB::ExecuteScalarSubqueriesMatcher::visit(DB::ASTFunction const&, std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e545dcb in /usr/bin/clickhouse
14. ./build_docker/./src/Interpreters/ExecuteScalarSubqueriesVisitor.cpp:0: DB::ExecuteScalarSubqueriesMatcher::visit(std::shared_ptr<DB::IAST>&, DB::ExecuteScalarSubqueriesMatcher::Data&) @ 0x1e540fbc in /usr/bin/clickhouse
15. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:78: DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::doVisit(std::shared_ptr<DB::IAST>&) @ 0x1e4ced8f in /usr/bin/clickhouse
16. ./build_docker/./src/Interpreters/InDepthNodeVisitor.h:64: void DB::InDepthNodeVisitor<DB::ExecuteScalarSubqueriesMatcher, true, false, std::shared_ptr<DB::IAST>>::visitChildren<false>(std::shared_ptr<DB::IAST>&) @ 0x1e4ceef9 in /usr/bin/clickhouse
17. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1551: DB::(anonymous namespace)::executeScalarSubqueries(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context const>, unsigned long, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, std::map<String, DB::Block, std::less<String>, std::allocator<std::pair<String const, DB::Block>>>&, bool, bool) @ 0x1e4c3574 in /usr/bin/clickhouse
18. ./build_docker/./src/Interpreters/TreeRewriter.cpp:0: DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>, bool, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>, std::unordered_map<String, String, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, String>>>) const @ 0x1e4beb6d in /usr/bin/clickhouse
19. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_4::operator()(bool) const @ 0x1e0be632 in /usr/bin/clickhouse
20. ./build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:0: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x1e0ba9ee in /usr/bin/clickhouse
21. ./build_docker/./contrib/llvm-project/libcxx/include/optional:260: DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e0b6726 in /usr/bin/clickhouse
22. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:257: DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::shared_ptr<DB::IAST> const&, std::vector<String, std::allocator<String>> const&) @ 0x1e180a1f in /usr/bin/clickhouse
23. ./build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152: DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x1e17dcc4 in /usr/bin/clickhouse
24. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:0: std::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) @ 0x1e06a3a9 in /usr/bin/clickhouse
25. ./build_docker/./src/Interpreters/InterpreterFactory.cpp:0: DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x1e0693a4 in /usr/bin/clickhouse
26. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x1e5a10f7 in /usr/bin/clickhouse
27. ./build_docker/./src/Interpreters/executeQuery.cpp:1160: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x1e59e6ec in /usr/bin/clickhouse
28. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f92faf4 in /usr/bin/clickhouse
29. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
30. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
31. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse

2024.03.20 06:01:22.036253 [ 756 ] {8b55c307-53b0-4edd-9ddd-b702ba689fb3} <Error> executeQuery: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:54222) (comment: 00175_counting_resources_in_subqueries.sql) (in query: SELECT count() FROM test.hits WHERE CounterID < (SELECT count() FROM test.hits WHERE NOT ignore(AdvEngineID));), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:01:22.045146 [ 756 ] {8b55c307-53b0-4edd-9ddd-b702ba689fb3} <Error> TCPHandler: Code: 158. DB::Exception: Limit for rows or bytes to read exceeded, max rows: 10.00 million, current rows: 13.12 million: While executing MergeTreeThread. (TOO_MANY_ROWS), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, String, String>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<String>::type, std::type_identity<String>::type>, char const*&, String&&, String&&) @ 0x1ccc2714 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:0: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc22c4 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:28.324466 [ 755 ] {612d35f0-7b00-4af9-8ba6-b4f7d399184d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.08973449 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:45838) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:28.324475 [ 2392 ] {54d05267-ea46-4cf0-b083-31908663affc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.229637599 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60706) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:28.324576 [ 2393 ] {3c06d613-36f3-4812-b8ed-f193a824e47f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.233534027 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43316) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:28.347343 [ 755 ] {612d35f0-7b00-4af9-8ba6-b4f7d399184d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.08973449 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:28.665746 [ 2392 ] {54d05267-ea46-4cf0-b083-31908663affc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.229637599 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:28.666224 [ 2393 ] {3c06d613-36f3-4812-b8ed-f193a824e47f} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:43316). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:28.667361 [ 2393 ] {3c06d613-36f3-4812-b8ed-f193a824e47f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.233534027 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:28.674405 [ 2393 ] {3c06d613-36f3-4812-b8ed-f193a824e47f} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:28.681804 [ 2392 ] {54d05267-ea46-4cf0-b083-31908663affc} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:29.928988 [ 755 ] {fb7484f9-14d0-488b-b0f0-4cdd230a2adb} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.147171595 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:45854) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:30.156907 [ 755 ] {fb7484f9-14d0-488b-b0f0-4cdd230a2adb} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.147171595 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:31.400032 [ 2393 ] {8191d2bb-ae13-496e-95fe-d8b5a9a945c7} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.004020623 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43320) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:31.400074 [ 2392 ] {fdf30621-db25-44d1-aa32-c72877f1fcde} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108022753 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60714) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:31.401376 [ 2394 ] {361edeb5-9bf9-4b93-ba1b-cd8b92010841} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.408903325 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37270) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:31.862323 [ 755 ] {85e51ece-dbce-4584-9bc5-f7d445e0c15b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.325277267 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:45864) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:32.102831 [ 755 ] {85e51ece-dbce-4584-9bc5-f7d445e0c15b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.325277267 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:32.254956 [ 1603 ] {0206b650-fe3e-4bb8-8e80-83a4ee04925d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.173195845 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43328) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:32.292140 [ 1603 ] {0206b650-fe3e-4bb8-8e80-83a4ee04925d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.173195845 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:32.334277 [ 1603 ] {0206b650-fe3e-4bb8-8e80-83a4ee04925d} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:33.862688 [ 1603 ] {c274d351-78e0-4a6b-9cfa-95c6336432ce} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.374201362 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:57578) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:34.156134 [ 1603 ] {c274d351-78e0-4a6b-9cfa-95c6336432ce} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.374201362 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:35.890457 [ 1603 ] {505c52ab-03ae-4592-80f8-dadca0c1857b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.270360469 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:57584) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.043220 [ 3160 ] {d584a234-9ee9-4ce8-9ca2-5295d68a4da6} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.197214973 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43992) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.150170 [ 3160 ] {d584a234-9ee9-4ce8-9ca2-5295d68a4da6} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.197214973 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.324269 [ 3157 ] {7b20dbc9-1a06-4486-b391-ceab404b2bd8} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.116022475 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51354) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.344883 [ 3158 ] {52aa7b7d-89a9-4b39-99db-251393a02c3c} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.076021668 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44058) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.399049 [ 3159 ] {8fc25ead-73a2-4cfd-8ead-0ee483421adc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.132022793 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:42114) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.628967 [ 3160 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:43992). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:36.629437 [ 1603 ] {505c52ab-03ae-4592-80f8-dadca0c1857b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.270360469 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.629790 [ 3157 ] {7b20dbc9-1a06-4486-b391-ceab404b2bd8} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.116022475 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.629790 [ 3159 ] {8fc25ead-73a2-4cfd-8ead-0ee483421adc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.132022793 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.632104 [ 3158 ] {52aa7b7d-89a9-4b39-99db-251393a02c3c} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.6]:9000 -> [::ffff:127.0.0.1]:44058). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:36.632808 [ 3158 ] {52aa7b7d-89a9-4b39-99db-251393a02c3c} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.076021668 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:36.636825 [ 3157 ] {7b20dbc9-1a06-4486-b391-ceab404b2bd8} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:36.638218 [ 3158 ] {52aa7b7d-89a9-4b39-99db-251393a02c3c} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:36.638239 [ 3159 ] {8fc25ead-73a2-4cfd-8ead-0ee483421adc} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:38.515826 [ 1603 ] {3d252386-8fd8-488f-a9cf-4f0d955b0595} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.475179958 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:57594) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:38.544016 [ 3162 ] {d389f96c-3e07-424c-bb82-f7c18913bb21} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.165585004 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44070) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:38.555618 [ 3162 ] {d389f96c-3e07-424c-bb82-f7c18913bb21} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.165585004 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:38.693873 [ 3162 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:44070). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:38.886598 [ 1603 ] {3d252386-8fd8-488f-a9cf-4f0d955b0595} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.475179958 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:39.108477 [ 3164 ] {8b6ede9f-7ce2-4d33-b13e-4bd00bf06755} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.052021005 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59380) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:39.127697 [ 3163 ] {40963440-aa76-4ce8-beaf-d853c51102b0} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108022123 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44008) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:39.193400 [ 3164 ] {8b6ede9f-7ce2-4d33-b13e-4bd00bf06755} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.052021005 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:39.222264 [ 3163 ] {40963440-aa76-4ce8-beaf-d853c51102b0} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:44008). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:39.223558 [ 3163 ] {40963440-aa76-4ce8-beaf-d853c51102b0} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108022123 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:39.223735 [ 3164 ] {8b6ede9f-7ce2-4d33-b13e-4bd00bf06755} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:39.250910 [ 3163 ] {40963440-aa76-4ce8-beaf-d853c51102b0} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:40.621663 [ 1603 ] {123b2af6-95ad-4027-8f3e-a81b1fda457f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.356889303 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:57608) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.681719 [ 3176 ] {3393c924-6326-4214-94a8-4a31a8f42f6d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.158781919 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44016) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.689628 [ 3176 ] {3393c924-6326-4214-94a8-4a31a8f42f6d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.158781919 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.703611 [ 3177 ] {a986cab4-2a74-4a1e-879b-3ec577d91501} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.147956829 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:42136) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.711479 [ 3177 ] {a986cab4-2a74-4a1e-879b-3ec577d91501} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.147956829 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.725360 [ 3177 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:42136). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:40.734951 [ 3162 ] {18b8fb26-3b75-4aa4-ab4b-9d1a2448795c} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.192023684 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59384) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.751524 [ 3162 ] {18b8fb26-3b75-4aa4-ab4b-9d1a2448795c} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.192023684 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.754944 [ 3176 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:44016). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:40.756341 [ 3162 ] {18b8fb26-3b75-4aa4-ab4b-9d1a2448795c} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:40.760873 [ 1603 ] {123b2af6-95ad-4027-8f3e-a81b1fda457f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.356889303 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.856631 [ 1276 ] {8eb22345-dd7a-4045-80ce-6cbdbdac4e1c} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 590.66 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:45124) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1ccc28f4 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc23ef in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.856627 [ 3173 ] {ff5477fe-fdb2-40af-9426-a8166f7d1d62} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.337560726 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44162) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.857254 [ 3174 ] {56ed216e-35fc-4d31-a004-ab9e78bbb0b4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.32397689 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43808) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.857267 [ 3175 ] {e78b947d-8ad3-4308-a9b2-344bbc96f2d3} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.32420884 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44076) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.873173 [ 3175 ] {e78b947d-8ad3-4308-a9b2-344bbc96f2d3} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.32420884 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.876337 [ 3173 ] {ff5477fe-fdb2-40af-9426-a8166f7d1d62} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.337560726 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.877917 [ 3175 ] {e78b947d-8ad3-4308-a9b2-344bbc96f2d3} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:40.879640 [ 3173 ] {ff5477fe-fdb2-40af-9426-a8166f7d1d62} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:40.882185 [ 3174 ] {56ed216e-35fc-4d31-a004-ab9e78bbb0b4} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:43808). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:40.882778 [ 3174 ] {56ed216e-35fc-4d31-a004-ab9e78bbb0b4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.32397689 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.885246 [ 3174 ] {56ed216e-35fc-4d31-a004-ab9e78bbb0b4} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:40.917580 [ 3148 ] {8e540839-ddcf-4baf-9940-320834790091} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.822617471 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:37272) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:40.976693 [ 3147 ] {60df30df-9ca5-4554-b472-ae15d5bd098a} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.889637862 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60546) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:41.798776 [ 3146 ] {b22ee3e7-f4a2-47ab-9597-b4eea8567ddd} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 10.697736242 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60728) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:41.948260 [ 1276 ] {8eb22345-dd7a-4045-80ce-6cbdbdac4e1c} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 590.66 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1ccc28f4 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc23ef in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.448680 [ 1603 ] {288ac67c-344c-40bf-99dc-80a521516bec} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.258088956 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:57612) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.518533 [ 3175 ] {192c11b9-6c1a-4165-b57c-9a4061f4afb3} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.00801992 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44078) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.558133 [ 3175 ] {192c11b9-6c1a-4165-b57c-9a4061f4afb3} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.00801992 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.744785 [ 3174 ] {bb0cf40e-71af-4738-89ac-11f324855d81} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348026622 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:42140) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.773955 [ 3162 ] {d200cd16-92d7-4daa-a9f4-3f6d7f10f56e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016020077 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43816) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.812979 [ 3174 ] {bb0cf40e-71af-4738-89ac-11f324855d81} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.348026622 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.868466 [ 3162 ] {d200cd16-92d7-4daa-a9f4-3f6d7f10f56e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.016020077 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.874637 [ 3175 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:44078). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:42.886312 [ 3176 ] {d98b17d1-0708-49b3-9eee-f23e99e77281} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100021733 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44174) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.887071 [ 3211 ] {e8c59b63-485b-4f82-83d0-0276e8be0d1e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.160022896 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44030) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.910090 [ 3213 ] {275ebb67-f91a-44ac-97d9-933d6dc1f867} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.136022419 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:56648) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.924687 [ 3211 ] {e8c59b63-485b-4f82-83d0-0276e8be0d1e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.160022896 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.934722 [ 3176 ] {d98b17d1-0708-49b3-9eee-f23e99e77281} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100021733 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.940391 [ 3213 ] {275ebb67-f91a-44ac-97d9-933d6dc1f867} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.136022419 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.947795 [ 3174 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:42140). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:42.956612 [ 3213 ] {275ebb67-f91a-44ac-97d9-933d6dc1f867} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:42.965848 [ 3212 ] {80538a10-491a-499c-aa29-00321595ccf5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.212023919 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:59390) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:42.991559 [ 3176 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:44174). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:42.994287 [ 3212 ] {80538a10-491a-499c-aa29-00321595ccf5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.212023919 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:43.019407 [ 3212 ] {80538a10-491a-499c-aa29-00321595ccf5} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:43.080124 [ 3211 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:44030). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:43.102713 [ 1603 ] {288ac67c-344c-40bf-99dc-80a521516bec} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.258088956 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:43.107775 [ 3162 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:43816). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:43.962958 [ 3151 ] {1128005c-3547-4d26-9b81-a600ab9576c0} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.080021888 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43772) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:44.414318 [ 755 ] {e49ae666-6b7f-42f7-99c7-569ec979b209} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.268025725 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:42108) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:44.975117 [ 3152 ] {b95ddf9f-f443-433f-b4b0-80b319b73f1f} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.076021808 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44140) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.019036 [ 1603 ] {3c9dcbfb-2f61-44e4-bec7-6161474ea567} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.520161079 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:57640) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT SearchPhrase AS k, count() AS c FROM remote('127.0.0.{1,2,3,4,5,6,7,8,9,10}', test.hits) GROUP BY k ORDER BY c DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.072072 [ 3150 ] {ffe59f56-1edf-4094-b489-f0611cd9691d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100022296 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43988) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.123751 [ 3153 ] {5375c945-552f-47f8-92c3-e110113b0873} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.092022129 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51350) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.188676 [ 3213 ] {b7446916-eb02-448b-8fa1-b5544a83a8e5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.300025481 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53132) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.200987 [ 3211 ] {01999b45-0bd7-4efb-b8b1-f86fe2956db4} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.340026265 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:35946) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.213700 [ 3212 ] {6aa1c2d3-5ba1-4d99-a119-baa16cc8d84e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.008019769 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54834) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.220510 [ 3296 ] {870c61c6-5497-43be-8576-b8413d5714b5} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.470435668 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:50620) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.227357 [ 3296 ] {870c61c6-5497-43be-8576-b8413d5714b5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.470435668 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.228712 [ 3298 ] {89e7cd91-c066-413b-9610-eb2bcebe7507} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.423270283 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:54952) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.229723 [ 3212 ] {6aa1c2d3-5ba1-4d99-a119-baa16cc8d84e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.008019769 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.236075 [ 3298 ] {89e7cd91-c066-413b-9610-eb2bcebe7507} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.423270283 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.236488 [ 3297 ] {5bf93221-57e7-42fe-83e1-5dff0ccad92b} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.437982154 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:60284) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.249606 [ 3297 ] {5bf93221-57e7-42fe-83e1-5dff0ccad92b} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.437982154 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.251585 [ 3212 ] {6aa1c2d3-5ba1-4d99-a119-baa16cc8d84e} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:45.252161 [ 3213 ] {b7446916-eb02-448b-8fa1-b5544a83a8e5} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.300025481 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.256018 [ 3296 ] {870c61c6-5497-43be-8576-b8413d5714b5} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:45.291521 [ 3211 ] {01999b45-0bd7-4efb-b8b1-f86fe2956db4} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.340026265 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.301360 [ 3211 ] {01999b45-0bd7-4efb-b8b1-f86fe2956db4} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:45.346115 [ 3174 ] {9f0edf54-3d36-42f3-b69f-605ccf7aede6} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.500029393 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:34522) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.356268 [ 3213 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:53132). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:45.363251 [ 3175 ] {ac5d2a49-084b-4fc7-9f60-0ce63871b21c} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.404027502 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:53308) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.423553 [ 3175 ] {ac5d2a49-084b-4fc7-9f60-0ce63871b21c} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.404027502 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.423585 [ 3174 ] {9f0edf54-3d36-42f3-b69f-605ccf7aede6} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.500029393 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.462444 [ 3175 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:53308). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:45.507677 [ 3297 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:60284). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:45.532540 [ 3298 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:54952). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:45.554364 [ 1603 ] {3c9dcbfb-2f61-44e4-bec7-6161474ea567} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.520161079 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.568224 [ 3174 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket ([::ffff:127.0.0.1]:34522). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x1ded457d in /usr/bin/clickhouse
3. ./build_docker/./src/IO/ReadBufferFromPocoSocket.cpp:0: DB::ReadBufferFromPocoSocket::nextImpl() @ 0x1ded3b50 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/ReadBuffer.h:67: DB::TCPHandler::runImpl() @ 0x1f92ef43 in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
6. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
7. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
9. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
11. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
12. ? @ 0x7fe874aa3609 in ?
13. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:45.844096 [ 3151 ] {1128005c-3547-4d26-9b81-a600ab9576c0} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:43772). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:45.845062 [ 3151 ] {1128005c-3547-4d26-9b81-a600ab9576c0} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.080021888 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:45.865280 [ 3151 ] {1128005c-3547-4d26-9b81-a600ab9576c0} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:46.658031 [ 3155 ] {a268dcbf-63fa-473e-89e8-935e3274f44d} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.136022879 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44142) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:46.845106 [ 3156 ] {319ff714-4e66-440e-b2b3-03c93b9395fd} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.132022803 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43788) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:49.416733 [ 3158 ] {357e6424-154a-4efd-a09d-80f84d613c80} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.388027758 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51362) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:49.451700 [ 3157 ] {c2b1bd43-b4c8-4959-b723-7b79cc193682} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.500029992 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:44146) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:49.901866 [ 3159 ] {67112fa3-a1b1-47a5-a022-2cb04a180bb7} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.356027118 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43796) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:49.916637 [ 3160 ] {c6568080-b2a6-4eaf-affc-8e7cb66247dc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.448028954 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:42128) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.142345 [ 3176 ] {4c79a8b4-fb84-4c4b-815e-14b9104a702e} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.236024239 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55404) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.158168 [ 3164 ] {fdd610ff-0a0b-40ee-9acf-480d0b63a189} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.09602178 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43562) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.160079 [ 3163 ] {e170add7-0f7a-48c6-bedc-8d4e23e81d14} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.156022973 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51364) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.164491 [ 3173 ] {ea0012df-b455-4269-ae60-0603d1d255b8} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 8.926273766 seconds, maximum: 1. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:51372) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.199303 [ 3177 ] {a8ff4e51-b8ae-4334-b361-9a1a31b715bc} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.072021181 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:43570) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.233536 [ 3162 ] {b4c4a448-1edb-4b95-a9bb-5f13fd9f01f3} <Error> executeQuery: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.192023378 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED) (version 23.3.19.34.altinitystable (altinity build)) (from [::ffff:127.0.0.1]:55334) (comment: 00090_thread_pool_deadlock.sh) (in query: SELECT `SearchPhrase` AS `k`, count() AS `c` FROM `test`.`hits` GROUP BY `k` ORDER BY `c` DESC LIMIT 10), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.764525 [ 3147 ] {60df30df-9ca5-4554-b472-ae15d5bd098a} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.889637862 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:50.771131 [ 3153 ] {5375c945-552f-47f8-92c3-e110113b0873} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.092022129 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:51.246839 [ 3153 ] {5375c945-552f-47f8-92c3-e110113b0873} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:51.265416 [ 3147 ] {60df30df-9ca5-4554-b472-ae15d5bd098a} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:51.662985 [ 2393 ] {8191d2bb-ae13-496e-95fe-d8b5a9a945c7} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:43320). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:51.665520 [ 2393 ] {8191d2bb-ae13-496e-95fe-d8b5a9a945c7} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.004020623 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:51.709568 [ 3155 ] {a268dcbf-63fa-473e-89e8-935e3274f44d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.136022879 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:51.710616 [ 2393 ] {8191d2bb-ae13-496e-95fe-d8b5a9a945c7} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:51.752954 [ 3155 ] {a268dcbf-63fa-473e-89e8-935e3274f44d} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:55.922294 [ 1276 ] {09542107-d442-4022-94ee-b94dfdf9769b} <Error> executeQuery: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 577.19 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES) (version 23.3.19.34.altinitystable (altinity build)) (from [::1]:45124) (comment: 00091_prewhere_two_conditions.sql) (in query: SELECT uniq(URL) FROM test.hits WHERE toTimeZone(EventTime, 'Asia/Dubai') >= '2014-03-20 00:00:00' AND URL != '' AND toTimeZone(EventTime, 'Asia/Dubai') < '2014-03-21 00:00:00';), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1ccc28f4 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc23ef in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:56.490428 [ 2394 ] {361edeb5-9bf9-4b93-ba1b-cd8b92010841} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 2.408903325 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:56.508341 [ 2394 ] {361edeb5-9bf9-4b93-ba1b-cd8b92010841} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:56.519679 [ 2392 ] {fdf30621-db25-44d1-aa32-c72877f1fcde} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:60714). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:56.520286 [ 2392 ] {fdf30621-db25-44d1-aa32-c72877f1fcde} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.108022753 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:56.529115 [ 2392 ] {fdf30621-db25-44d1-aa32-c72877f1fcde} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:56.610666 [ 3160 ] {c6568080-b2a6-4eaf-affc-8e7cb66247dc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.448028954 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:56.616426 [ 3160 ] {c6568080-b2a6-4eaf-affc-8e7cb66247dc} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:56.937121 [ 3176 ] {4c79a8b4-fb84-4c4b-815e-14b9104a702e} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.236024239 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:56.942700 [ 3176 ] {4c79a8b4-fb84-4c4b-815e-14b9104a702e} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:57.094847 [ 1276 ] {09542107-d442-4022-94ee-b94dfdf9769b} <Error> TCPHandler: Code: 307. DB::Exception: Limit for rows or bytes to read exceeded, max bytes: 572.20 MiB, current bytes: 577.19 MiB: While executing MergeTreeThread. (TOO_MANY_BYTES), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<char const*&, ReadableSize, ReadableSize>(int, FormatStringHelperImpl<std::type_identity<char const*&>::type, std::type_identity<ReadableSize>::type, std::type_identity<ReadableSize>::type>, char const*&, ReadableSize&&, ReadableSize&&) @ 0x1ccc28f4 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/SizeLimits.cpp:36: DB::SizeLimits::check(unsigned long, unsigned long, char const*, int, int) const @ 0x1ccc23ef in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:97: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca83bc in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:57.521233 [ 3157 ] {c2b1bd43-b4c8-4959-b723-7b79cc193682} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.500029992 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:57.521997 [ 3157 ] {c2b1bd43-b4c8-4959-b723-7b79cc193682} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:58.560631 [ 3146 ] {b22ee3e7-f4a2-47ab-9597-b4eea8567ddd} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.2]:9000 -> [::ffff:127.0.0.1]:60728). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:58.561380 [ 3146 ] {b22ee3e7-f4a2-47ab-9597-b4eea8567ddd} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 10.697736242 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:58.575780 [ 3146 ] {b22ee3e7-f4a2-47ab-9597-b4eea8567ddd} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:58.619267 [ 3177 ] {a8ff4e51-b8ae-4334-b361-9a1a31b715bc} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.072021181 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:58.629661 [ 3177 ] {a8ff4e51-b8ae-4334-b361-9a1a31b715bc} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:58.944216 [ 3150 ] {ffe59f56-1edf-4094-b489-f0611cd9691d} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.100022296 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:58.945202 [ 3150 ] {ffe59f56-1edf-4094-b489-f0611cd9691d} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:59.152183 [ 3152 ] {b95ddf9f-f443-433f-b4b0-80b319b73f1f} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.4]:9000 -> [::ffff:127.0.0.1]:44140). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:59.152904 [ 3152 ] {b95ddf9f-f443-433f-b4b0-80b319b73f1f} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.076021808 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:59.160007 [ 3152 ] {b95ddf9f-f443-433f-b4b0-80b319b73f1f} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:59.477207 [ 755 ] {e49ae666-6b7f-42f7-99c7-569ec979b209} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.268025725 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:59.492410 [ 755 ] {e49ae666-6b7f-42f7-99c7-569ec979b209} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:11:59.854945 [ 3148 ] {8e540839-ddcf-4baf-9940-320834790091} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:37272). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:11:59.855455 [ 3148 ] {8e540839-ddcf-4baf-9940-320834790091} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 9.822617471 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:11:59.878066 [ 3148 ] {8e540839-ddcf-4baf-9940-320834790091} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:12:01.092414 [ 3156 ] {319ff714-4e66-440e-b2b3-03c93b9395fd} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:43788). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:12:01.101191 [ 3156 ] {319ff714-4e66-440e-b2b3-03c93b9395fd} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.132022803 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:12:01.109434 [ 3156 ] {319ff714-4e66-440e-b2b3-03c93b9395fd} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:12:01.171591 [ 3164 ] {fdd610ff-0a0b-40ee-9acf-480d0b63a189} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.09602178 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:12:01.174221 [ 3164 ] {fdd610ff-0a0b-40ee-9acf-480d0b63a189} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:12:01.187705 [ 3162 ] {b4c4a448-1edb-4b95-a9bb-5f13fd9f01f3} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.192023378 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:12:01.188786 [ 3162 ] {b4c4a448-1edb-4b95-a9bb-5f13fd9f01f3} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:12:01.620608 [ 3159 ] {67112fa3-a1b1-47a5-a022-2cb04a180bb7} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.1]:9000 -> [::ffff:127.0.0.1]:43796). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:12:01.621363 [ 3159 ] {67112fa3-a1b1-47a5-a022-2cb04a180bb7} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.356027118 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:12:01.628714 [ 3159 ] {67112fa3-a1b1-47a5-a022-2cb04a180bb7} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:12:01.844344 [ 3158 ] {357e6424-154a-4efd-a09d-80f84d613c80} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:51362). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:12:01.844859 [ 3158 ] {357e6424-154a-4efd-a09d-80f84d613c80} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.388027758 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:12:01.845934 [ 3158 ] {357e6424-154a-4efd-a09d-80f84d613c80} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:12:02.454178 [ 3173 ] {ea0012df-b455-4269-ae60-0603d1d255b8} <Error> TCPHandler: Can't send logs to client: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:127.0.0.3]:9000 -> [::ffff:127.0.0.1]:51372). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x1ded5d0e in /usr/bin/clickhouse
3. ./build_docker/./src/IO/WriteBufferFromPocoSocket.cpp:0: DB::WriteBufferFromPocoSocket::nextImpl() @ 0x1ded52f8 in /usr/bin/clickhouse
4. ./build_docker/./src/IO/BufferBase.h:39: DB::TCPHandler::sendLogData(DB::Block const&) @ 0x1f94665c in /usr/bin/clickhouse
5. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::sendLogs() @ 0x1f93ac28 in /usr/bin/clickhouse
6. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x1f931430 in /usr/bin/clickhouse
7. ./build_docker/./src/Server/TCPHandler.cpp:2079: DB::TCPHandler::run() @ 0x1f946a48 in /usr/bin/clickhouse
8. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x236e4883 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x236e50f3 in /usr/bin/clickhouse
10. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:202: Poco::PooledThread::run() @ 0x2394069a in /usr/bin/clickhouse
11. ./build_docker/./base/poco/Foundation/src/Thread.cpp:46: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x2393ea50 in /usr/bin/clickhouse
12. ./build_docker/./base/poco/Foundation/include/Poco/SharedPtr.h:231: Poco::ThreadImpl::runnableEntry(void*) @ 0x2393d068 in /usr/bin/clickhouse
13. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
14. ? @ 0x7fe874aa3609 in ?
15. __clone @ 0x7fe8749c8353 in ?
 (version 23.3.19.34.altinitystable (altinity build))
2024.03.20 06:12:02.454637 [ 3173 ] {ea0012df-b455-4269-ae60-0603d1d255b8} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 8.926273766 seconds, maximum: 1. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/Interpreters/ProcessList.cpp:0: DB::QueryStatus::checkTimeLimit() @ 0x1e235afd in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:192: DB::PipelineExecutor::finalizeExecution() @ 0x1f9a90c3 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:0: DB::PipelineExecutor::execute(unsigned long) @ 0x1f9a8d29 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9bd916 in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:12:02.456877 [ 3173 ] {ea0012df-b455-4269-ae60-0603d1d255b8} <Warning> TCPHandler: Client has gone away.
2024.03.20 06:12:02.846225 [ 3163 ] {e170add7-0f7a-48c6-bedc-8d4e23e81d14} <Error> TCPHandler: Code: 159. DB::Exception: Timeout exceeded: elapsed 1.156022973 seconds, maximum: 1: While executing MergeTreeThread. (TIMEOUT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:0: Poco::Exception::Exception(String const&, int) @ 0x238a569e in /usr/bin/clickhouse
1. ./build_docker/./src/Common/Exception.cpp:91: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x146e0ab8 in /usr/bin/clickhouse
2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception<double, double>(int, FormatStringHelperImpl<std::type_identity<double>::type, std::type_identity<double>::type>, double&&, double&&) @ 0x1cca9705 in /usr/bin/clickhouse
3. ./build_docker/./src/QueryPipeline/ExecutionSpeedLimits.cpp:110: DB::ExecutionSpeedLimits::checkTimeLimit(Stopwatch const&, DB::OverflowMode) const @ 0x1cca9231 in /usr/bin/clickhouse
4. ./build_docker/./src/QueryPipeline/ReadProgressCallback.cpp:0: DB::ReadProgressCallback::onProgress(unsigned long, unsigned long, std::list<DB::StorageLimits, std::allocator<DB::StorageLimits>> const&) @ 0x1cca80cb in /usr/bin/clickhouse
5. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:59: DB::ExecutionThreadContext::executeTask() @ 0x1f9b8fa2 in /usr/bin/clickhouse
6. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:246: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x1f9a9c41 in /usr/bin/clickhouse
7. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:345: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x1f9ab4ee in /usr/bin/clickhouse
8. ./build_docker/./base/base/../base/wide_integer_impl.h:789: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x147fa8e8 in /usr/bin/clickhouse
9. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, long, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x148003b2 in /usr/bin/clickhouse
10. __tsan_thread_start_func @ 0xc6017e7 in /usr/bin/clickhouse
11. ? @ 0x7fe874aa3609 in ?
12. __clone @ 0x7fe8749c8353 in ?

2024.03.20 06:12:02.847925 [ 3163 ] {e170add7-0f7a-48c6-bedc-8d4e23e81d14} <Warning> TCPHandler: Client has gone away.